Production Monitoring in OpenTelemetry Demo

Monitoring Overview

In a production environment, effective monitoring is crucial for maintaining the health of services. The OpenTelemetry Demo project employs a variety of techniques to instrument, trace, and monitor services built with different programming languages. This document focuses on Go and Java service monitoring.

1. Monitoring Go Services

The OpenTelemetry Go implementation enables us to instrument our services seamlessly. When building a Go service, ensure the following module is included to enable OpenTelemetry tracing:

module github.com/open-telemetry/opentelemetry-demo/src/checkoutservice

Use the OpenTelemetry SDK to set up the tracer and record spans:

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/sdk/resource"
    "go.opentelemetry.io/otel/sdk/tracing"
)

// SetupTracer initializes the OpenTelemetry tracer
func SetupTracer() {
    rs, err := resource.New(context.Background(),
        resource.WithAttributes(
            semconv.ServiceNameKey.String("checkoutservice"),
        ),
    )
    if err != nil {
        log.Fatal(err)
    }

    tracerProvider := tracing.NewTracerProvider(tracing.WithResource(rs))
    otel.SetTracerProvider(tracerProvider)
}

With instrumentation set up, add traces in your endpoints. For example, in a checkout handler:

func CheckoutHandler(w http.ResponseWriter, r *http.Request) {
    ctx, span := otel.Tracer("checkout").Start(r.Context(), "CheckoutHandler")
    defer span.End()

    // business logic for checkout
}

2. Monitoring Java Services

To monitor the Java services, the Dockerfile provided in the project uses an OpenTelemetry Java agent for automatic instrumentation. Below is the critical section of the Dockerfile that sets up the Java service:

FROM --platform=${BUILDPLATFORM} gradle:8-jdk17 AS builder

WORKDIR /usr/src/app/

COPY ./src/frauddetectionservice/ ./
COPY ./pb/ ./src/main/proto/
RUN gradle shadowJar

FROM gcr.io/distroless/java17-debian11

ARG OTEL_JAVA_AGENT_VERSION
WORKDIR /usr/src/app/

COPY --from=builder /usr/src/app/build/libs/frauddetectionservice-1.0-all.jar ./
ADD --chmod=644 https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v$OTEL_JAVA_AGENT_VERSION/opentelemetry-javaagent.jar /app/opentelemetry-javaagent.jar
ENV JAVA_TOOL_OPTIONS=-javaagent:/app/opentelemetry-javaagent.jar

ENTRYPOINT [ "java", "-jar", "frauddetectionservice-1.0-all.jar" ]

The above configuration assumes the OpenTelemetry Java agent is available at runtime and instrument any library calls automatically. You can customize the agent behavior through various configuration options, like endpoint and service name.

3. Integration into CI/CD Pipeline

Ensure that the deployment process is also instrumented. Utilize a Makefile to define the build process. Below is a simplistic example of a Makefile that can be utilized for Go and Java services:

.PHONY: build

build:
    docker build -t my-java-app .

deploy:
    kubectl apply -f deployment.yaml

Run the deployment with monitoring in mind, ensuring environment variables are set up correctly.

4. Viewing Traces and Metrics

To visualize and analyze the collected traces and metrics, integrate with a tracing backend such as Jaeger or Zipkin. Ensure that your services are configured to send data to these backends.

In your configuration, set the endpoint to your Jaeger or Zipkin service:

traceExporter, err := jaegercfg.New(
    jaegercfg.WithAgentHost("jaeger-agent-host"),
    jaegercfg.WithAgentPort("6831"),
)

For Java, you can set environment variables for the OpenTelemetry agent to direct data appropriately:

OTEL_EXPORTER_JAEGER_ENDPOINT=http://jaeger-collector:14268/api/traces

Consult the OpenTelemetry documentation for specific environmental configurations tailored to your infrastructure needs.

By adopting the above practices and configurations, production monitoring in the OpenTelemetry Demo project will ensure robustness and maintainability.

Source: OpenTelemetry Demo GitHub Repository