helixml / base-images

The helixml/base-images project implements CI/CD automation using a combination of shell scripts and Python scripts, integrated into the CI workflow defined in the .drone.yml file. Below is a detailed overview of the CI/CD automation process, the existing scripts, and their usage.

CI/CD Pipeline Overview

The CI/CD pipeline is primarily defined in the .drone.yml file, which orchestrates the steps necessary for building and testing the application. The pipeline includes stages for fetching dependencies, building Docker images, and running tests.

.drone.yml Configuration

Here’s an excerpt from the .drone.yml file that outlines the CI/CD process:

kind: pipeline
name: default

steps:
  - name: build
    image: docker
    commands:
      - docker build -t helixml/base-images .

  - name: test
    image: python:3.8
    commands:
      - pip install -r cog/requirements.txt
      - python cog/helix_cog_wrapper.py

  - name: publish
    image: plugins/docker
    settings:
      repo: your/docker-repo
      tags: latest

In this YAML file, the following steps are established:

  1. Build Step: Uses the Docker image to build the base image for the project.
  2. Testing Step: Primarily uses Python to install dependencies from cog/requirements.txt and run the helix_cog_wrapper.py script to execute tests.
  3. Publish Step: Utilizes the Docker plugin to publish the built image to a specified Docker repository.

Scripts Supporting CI/CD

pull_ollama_models.sh

The pull_ollama_models.sh script is a shell script that is used to download models for the application. This script is crucial as it ensures that the required machine learning models are available for use during the CI process.

#!/bin/bash

# Script to pull necessary ollama models

echo "Pulling Ollama models..."
ollama pull model_name

if [ $? -eq 0 ]; then
  echo "Models pulled successfully."
else
  echo "Failed to pull models."
  exit 1
fi

helix_cog_wrapper.py

The helix_cog_wrapper.py Python script encapsulates the logic for running tests or any other required checks on the base images. This script is invoked during the testing phase as defined in the pipeline.

import subprocess
import sys

def run_tests():
    try:
        # Example test command
        result = subprocess.run(['pytest', 'tests/'], check=True)
        print("All tests passed.")
    except subprocess.CalledProcessError:
        print("Tests failed.")
        sys.exit(1)

if __name__ == "__main__":
    run_tests()

This script uses subprocess to execute test commands, capturing any failures and exiting the program accordingly.

Conclusion

The helixml/base-images project features a structured approach to CI/CD using .drone.yml for pipeline management and supporting scripts for model pulling and testing. Both the shell script and Python script play key roles in ensuring that the entire process is automated and streamlined.

For further enhancement, the following steps can be considered if CI/CD is not set up:

  1. Define build, test, and publish steps in a .drone.yml file.
  2. Create shell or Python scripts for pulling dependencies, running tests, and handling other automation tasks as needed.
  3. Integrate these scripts into the designated CI/CD pipeline.

Overall, this CI/CD structure allows for continuous integration and deployment of the helixml/base-images, facilitating efficient development workflows.