Overview

This document outlines the procedure for scaling the OpenTelemetry demo in a production environment. The examples provided illustrate how the application can be built and deployed with considerations for horizontal scaling, resource management, and observability, which are essential for robust production deployments.

Scaling Strategy

1. Containerization with Docker

Utilizing Docker for containerization allows for consistent deployment units across various environments. The Dockerfile provided below is designed to build the project using Gradle and set up the Java application runtime with OpenTelemetry instrumentation.

FROM --platform=${BUILDPLATFORM} gradle:8-jdk17 AS builder

WORKDIR /usr/src/app/

COPY ./src/frauddetectionservice/ ./
COPY ./pb/ ./src/main/proto/
RUN gradle shadowJar

# -----------------------------------------------------------------------------

FROM gcr.io/distroless/java17-debian11

ARG OTEL_JAVA_AGENT_VERSION
WORKDIR /usr/src/app/

COPY --from=builder /usr/src/app/build/libs/frauddetectionservice-1.0-all.jar ./
ADD --chmod=644 https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v$OTEL_JAVA_AGENT_VERSION/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar
ENV JAVA_TOOL_OPTIONS=-javaagent:/app/opentelemetry-javaagent.jar

ENTRYPOINT [ "java", "-jar", "frauddetectionservice-1.0-all.jar" ]

In the above Dockerfile:

• The first stage builds the application using the Gradle image.
• The second stage uses a distroless image which minimizes the size and attack surface of the deployed image.
• OpenTelemetry’s Java agent is included and configured to monitor the application.

2. Building The Application

For Go modules, ensure no vendor directories are included. When building the application, reference the Go module specifically linked to the checkout service.

cd src/checkoutservice
go build

Specify the module path to control dependencies effectively and to assure consistency across environments.

3. Resource Management for Scaling

When deploying applications in a container orchestrator such as Kubernetes, define resource requests and limits. Here is a sample YAML configuration for the frauddetectionservice:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frauddetectionservice
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frauddetectionservice
  template:
    metadata:
      labels:
        app: frauddetectionservice
    spec:
      containers:
      - name: frauddetectionservice
        image: <your-image-repository>/frauddetectionservice:latest
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"

4. Implementing Horizontal Pod Autoscaling (HPA)

To handle varying loads, implement HPA. This adjusts the number of pod replicas in response to observed CPU usage:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: frauddetectionservice
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frauddetectionservice
  minReplicas: 1
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 75

5. Use of OpenTelemetry for Monitoring and Performance Tuning

Integrating OpenTelemetry not only aids in monitoring but also helps in fine-tuning the performance of services. Ensure that tracing and metrics collection are configured as per your application needs. The instrumentation can be added via the Java agent discussed earlier in the Dockerfile.

Conclusion

Effective scaling of the OpenTelemetry demo involves utilizing containerization, careful resource management, and leveraging Kubernetes features such as HPA alongside OpenTelemetry for observability. By following these steps, production deployments can achieve higher efficiency and responsiveness to load changes.

Source: OpenTelemetry Demo Repository