CI/CD (Continuous Integration/Continuous Delivery) for Anthias

Motivation

The CI/CD pipeline implemented in Anthias streamlines development, automates testing and deployment, and enables rapid iteration on new features.

Workflows

The following workflows are implemented in Anthias:

• Python Linting:
  • Why: Ensures code quality and consistency.
  • How: Uses flake8 linter.
  • Locally:
    • Install act: https://github.com/nektos/act
    • Run:

      $ act -W .github/workflows/python-lint.yaml
                

  • CI:
    • Runs on every push and pull request.
    • .github/workflows/python-lint.yaml
• Unit Tests:
  • Why: Ensures code functionality.
  • How: Uses pytest framework.
  • Locally:
    1. Build and start containers:

       $ poetry run python tools/image_builder \
                 --dockerfiles-only \
                 --disable-cache-mounts \
                 --service celery \
                 --service redis \
                 --service test
                 $ docker compose \
                 -f docker-compose.test.yml up -d --build
                 

    2. Run tests:

       $ docker compose \
                 -f docker-compose.test.yml \
                 exec anthias-test bash ./bin/prepare_test_environment.sh -s
                 
                 # Integration and non-integration tests should be run separately as the
                 # former doesn't run as expected when run together with the latter.
                 
                 $ docker compose \
                 -f docker-compose.test.yml \
                 exec anthias-test ./manage.py test --exclude-tag=integration
                 
                 $ docker compose \
                 -f docker-compose.test.yml \
                 exec anthias-test ./manage.py test --tag=integration
                 

  • CI:
    • Runs on every push and pull request to master.
    • .github/workflows/run-tests.yaml
• Code Quality Analysis:
  • Why: Identifies potential security vulnerabilities and code quality issues.
  • How: Uses CodeQL.
  • CI:
    • Runs on every push and pull request to master and production.
    • .github/workflows/codeql-analysis.yaml
• Balena Disk Image Build:
  • Why: Creates disk images for various Balena boards.
  • How: Uses Docker and Balena tools.
  • CI:
    • Runs on demand via workflow dispatch.
    • .github/workflows/build-balena-disk-image.yaml
• Docker Image Build:
  • Why: Creates Docker images for various services.
  • How: Uses Docker.
  • CI:
    • Runs on demand via workflow dispatch.
    • .github/workflows/docker-build.yaml
• Release Tagging:
  • Why: Creates release tags for version control.
  • How: Uses Git tags.
  • Locally:
    • Check latest release:

      $ git pull
                $ git tag
                

    • Create new release:

      $ git tag -a v0.18.7 -m "Test new automated disk images"
                

    • Push release:

      $ git push origin v0.18.7
                

Development Server

• Why: Allows developers to test the application locally.
• How:
  • See Developer Documentation for details.

Testing

• Why: Ensures application functionality.
• How:
  • Unit Tests: Use pytest framework.
  • Manual Testing: Use the provided QA Checklist.
• CI:
  • Unit tests are run on every push and pull request to master.

Code Examples

• lib/utils.py:

  def is_ci():
                """
                Returns True when run on CI.
                """
                return string_to_bool(os.getenv('CI', False))
            

• .github/workflows/docker-test.yaml:

  - name: Upload coverage reports to Codecov
                uses: codecov/codecov-action@v5
                env:
                  CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
            
              - name: Stop the test container
                run: |
                  docker compose -f docker-compose.test.yml down
            

• .github/workflows/docker-build.yaml:

  - uses: balena-io/deploy-to-balena-action@master
                id: build
                continue-on-error: true
                with:
                  balena_token: ${{ secrets.BALENA_TOKEN }}
                  fleet: screenly_ose/anthias-${{ matrix.board }}
                  source: balena-deploy
            
              - name: Sleep random sleep before retry
                if: ${{ failure() && steps.build.conclusion == 'failure' }}
                run: |
                  sleep $((120 + RANDOM % 900));
            
              # Balena deploy often fails with 'ESOCKETTIMEDOUT'.
              # This adds some retry logic.
              - uses: balena-io/deploy-to-balena-action@master
                id: build-retry
                if: ${{ failure() && steps.build.conclusion == 'failure' }}
                with:
                  balena_token: ${{ secrets.BALENA_TOKEN }}
                  fleet: screenly_ose/anthias-${{ matrix.board }}
                  source: balena-deploy
            

References

• .gitignore
• README.md
• docs/developer-documentation.md
• docs/qa-checklist.md
• license
• docker-compose.balena.dev.yml.tmpl
• docker/dockerfile.celery.j2
• lib/utils.py
• tests/test_celery_tasks.py

          ## Top-Level Directory Explanations
          
          <a class='local-link directory-link' data-ref=".github/" href="#.github/">.github/</a> - This directory contains GitHub-specific configuration files and workflows for the project.
          
          <a class='local-link directory-link' data-ref=".github/workflows/" href="#.github/workflows/">.github/workflows/</a> - This directory contains YAML files defining continuous integration and deployment workflows for GitHub Actions.
          
          <a class='local-link directory-link' data-ref="ansible/" href="#ansible/">ansible/</a> - Ansible is an open-source configuration management and automation tool. This directory contains Ansible playbooks and roles for managing and configuring the project.
          
          <a class='local-link directory-link' data-ref="ansible/roles/" href="#ansible/roles/">ansible/roles/</a> - This directory contains Ansible roles, which are reusable collections of tasks and configurations for managing specific aspects of a system.
          
          <a class='local-link directory-link' data-ref="ansible/roles/screenly/" href="#ansible/roles/screenly/">ansible/roles/screenly/</a> - This role is specific to the Screenly project and likely contains configurations and tasks related to it.
          
          <a class='local-link directory-link' data-ref="ansible/roles/system/" href="#ansible/roles/system/">ansible/roles/system/</a> - This role manages system-level configurations.
          
          <a class='local-link directory-link' data-ref="anthias_app/" href="#anthias_app/">anthias_app/</a> - This directory contains the main application codebase for the project, likely written in Django.
          
          <a class='local-link directory-link' data-ref="anthias_app/migrations/" href="#anthias_app/migrations/">anthias_app/migrations/</a> - This directory contains database migration files for the Django application.
          
          <a class='local-link directory-link' data-ref="anthias_django/" href="#anthias_django/">anthias_django/</a> - This directory may contain additional Django-specific configuration files and code.
          
          <a class='local-link directory-link' data-ref="api/" href="#api/">api/</a> - This directory contains the API codebase for the project.
          
          <a class='local-link directory-link' data-ref="bin/" href="#bin/">bin/</a> - This directory contains executable scripts for the project.
          
          <a class='local-link directory-link' data-ref="lib/" href="#lib/">lib/</a> - This directory contains reusable Python modules and libraries for the project.
          
          <a class='local-link directory-link' data-ref="static/" href="#static/">static/</a> - This directory contains static files, such as images, CSS, and JavaScript, that are served directly to the user by the web server.
          
          <a class='local-link directory-link' data-ref="static/spec/" href="#static/spec/">static/spec/</a> - This directory contains test files for the static files.
          
          <a class='local-link directory-link' data-ref="static/spec/jasmine/" href="#static/spec/jasmine/">static/spec/jasmine/</a> - This directory contains Jasmine test files.
          
          <a class='local-link directory-link' data-ref="templates/" href="#templates/">templates/</a> - This directory contains HTML templates used to render dynamic content.
          
          <a class='local-link directory-link' data-ref="tests/" href="#tests/">tests/</a> - This directory contains test files for the project.
          
          <a class='local-link directory-link' data-ref="tools/" href="#tools/">tools/</a> - This directory contains tools and scripts used to develop and maintain the project.
          
          <a class='local-link directory-link' data-ref="tools/image_builder/" href="#tools/image_builder/">tools/image_builder/</a> - This tool likely builds and optimizes images for the project.
          
          <a class='local-link directory-link' data-ref="website/" href="#website/">website/</a> - This directory contains the website codebase.
          
          <a class='local-link directory-link' data-ref="website/bin/" href="#website/bin/">website/bin/</a> - This directory contains website executable scripts.
          
          <a class='local-link directory-link' data-ref="webview/" href="#webview/">webview/</a> - This directory likely contains configuration files and code for a webview component.