HelixML Architecture - helixml/helix

HelixML Architecture

The Helix project (https://github.com/helixml/helix/) utilizes several key technologies and dependencies, including Go, Docker, Git, PostgreSQL, and others. This document will explain the architecture of HelixML, focusing on its main components and their interactions.

The Big Picture

HelixML is designed as a microservices-based architecture, with each service having a specific role and communicating with others through APIs. The main services in HelixML are:

  1. CurveBS: A scalable and reliable block storage service built on top of PostgreSQL. It provides data storage and caching functionality. More details can be found in the Curve documentation (https://github.com/opencurve/curve).
  2. CurveAdm: A tool for deploying and managing Curve clusters. It supports the deployment of CurveBS and CurveFS, with scaling, upgrading, and other functions under development. Refer to the CurveAdm documentation (https://curvebs.readthedocs.io/en/latest/) for more information.
  3. API Gateway: An entry point for external services to interact with HelixML. It uses Gorilla Router for routing requests and JWT for authentication.
  4. Go Mocking Framework: A library for generating mock objects during testing.
  5. Gorilla Websocket: A WebSocket library for real-time communication between services.
  6. Material UI: A UI library for building web interfaces.

Design Philosophy

HelixML follows a few design principles:

  1. Modularity: Each service is designed to be independent and have minimal dependencies on other services. This allows for better scalability and maintainability.
  2. Automation: HelixML uses tools like Docker and Kubernetes to automate deployment, scaling, and management of services.
  3. Security: Services communicate through secure APIs, using JWT for authentication and TLS for encryption.

Programming Languages

HelixML primarily uses Go as its programming language, taking advantage of its simplicity, performance, and strong support for concurrency. The project also uses HCL2 for Packer templates and SQL for database interactions.

Go Programming Language

Go, also known as Golang, is a statically typed, compiled language developed by Google. It is designed for simplicity, concurrency, and productivity. HelixML uses Go for its main services, such as CurveBS and CurveAdm.

Docker

Docker is a platform for developing, shipping, and running applications using container technology. HelixML uses Docker to package and distribute its services, ensuring consistent execution across different environments.

Git

Git is a distributed version control system that HelixML uses for managing its source code. The project’s main repository is hosted on GitHub (https://github.com/helixml/helix/).

PostgreSQL

PostgreSQL is a powerful, open-source object-relational database system. HelixML uses PostgreSQL as the primary data storage for CurveBS, providing data storage and caching functionality.

Go Mocking Framework

The Go Mocking Framework is a library for generating mock objects during testing. It allows developers to create test doubles for interfaces, making it easier to test code in isolation.

Gorilla Router

Gorilla Router is a HTTP request router and dispatcher for building Go web servers. HelixML uses Gorilla Router to route incoming requests to the appropriate handler functions.

Gorilla Websocket

Gorilla Websocket is a WebSocket library for building real-time, bidirectional communication between web clients and servers. HelixML uses Gorilla Websocket for real-time data transfer between services.

Material UI

Material UI is a popular React UI framework that follows Google’s Material Design guidelines. HelixML uses Material UI to build its web interfaces, ensuring a consistent and modern look and feel.

JWT

JSON Web Tokens (JWT) are a compact, URL-safe means of representing claims to be transferred between two parties. HelixML uses JWT for authentication, allowing services to securely identify clients and grant access to protected resources.

Go CMP

Go CMP (https://github.com/helixml/cmp) is a custom HelixML component for managing configuration and metadata. It is used by CurveAdm to manage Curve cluster configurations.

Examples

CurveBS Deployment

To deploy CurveBS using CurveAdm, follow these steps:

  1. Install CurveAdm:
curl -L https://raw.githubusercontent.com/curvebs/curveadm/master/install.sh | sh
  1. Create a cluster topology file for standalone deployment:
# curvebs_standalone.yaml
apiVersion: curveadm.io/v1alpha1
kind: ClusterTopology
metadata:
name: curvebs-standalone
spec:
replicas: 1
zone:
- name: zone1
replicas: 1
meta:
region: us-west-1
  1. Deploy CurveBS:
curveadm deploy -f curvebs_standalone.yaml

Packer HCL2 Template

Here’s an example of a Packer HCL2 template that builds a Docker image:

packer {
required_plugins {
docker = {
version = ">= 1.0.0"
}
}
}

source "docker" "my-docker-image" {
image = "my-base-image:latest"
commit = true

provisioner "shell" {
inline = [
"echo 'Hello, World!'"
]
}
}

build {
sources = [
"source.docker.my-docker-image"
]
}

Conclusion

HelixML is a microservices-based architecture that utilizes various technologies and dependencies to provide a scalable and reliable block storage service. By following design principles such as modularity, automation, and security, HelixML ensures a robust and maintainable system.