This documentation outlines the steps and methods used for monitoring the Sourcegraph/Zoekt project in a production environment. It covers critical components and techniques that developers can utilize for effective observability and performance tracking.

Monitoring Setup

1. Instrumentation in Go Code

To effectively monitor the performance and health of the Zoekt service, it is essential to instrument the code with appropriate metrics. The Prometheus Go client library is commonly used for exposing metrics.

Add the following dependencies to your go.mod file:

require (
    github.com/prometheus/client_golang v1.11.0
)

2. Exposing Metrics

You can expose metrics in your application by creating a new HTTP handler. This handler will utilize the Prometheus library to provide metrics.

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "net/http"
)

var (
    httpRequestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "handler"},
    )
)

func init() {
    prometheus.MustRegister(httpRequestsTotal)
}

func myHandler(w http.ResponseWriter, r *http.Request) {
    httpRequestsTotal.WithLabelValues(r.Method, "/myHandler").Inc()
    // Your handler logic
}

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/myHandler", myHandler)
    http.ListenAndServe(":8080", nil)
}

In this example, a counter is created to track the total number of HTTP requests to a specific handler. This metric can be scraped by Prometheus by accessing the /metrics endpoint.

3. Setting Up Prometheus

Set up Prometheus to scrape metrics by configuring the prometheus.yml file. Add the following job definition:

scrape_configs:
  - job_name: 'zoekt'
    static_configs:
      - targets: ['localhost:8080']

Replace localhost:8080 with the actual host and port where Zoekt is running.

4. Logging

Using structured logging enhances monitoring by making logs easier to parse and analyze. Incorporate a package such as logrus or zap for structured logging.

Here is an example using logrus:

package main

import (
    "github.com/sirupsen/logrus"
)

func main() {
    logger := logrus.New()
    logger.SetFormatter(&logrus.JSONFormatter{})
    logger.SetLevel(logrus.InfoLevel)

    logger.Info("Starting the service")
}

Ensure logs are written to a centralized logging system for aggregation and analysis. Use tools like Elasticsearch, Fluentd, and Kibana (EFK) or the ELK stack for impressive log management.

5. Health Checks

Implement health checks to ensure the service is operational. In the Zoekt service, it is common to have a /health endpoint that can be queried.

func healthCheck(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("OK"))
}

func main() {
    http.HandleFunc("/health", healthCheck)
    http.ListenAndServe(":8080", nil)
}

Prometheus can also be configured to scrape this endpoint to check the health of the service.

6. Alerting

Set up alerting rules in Prometheus to notify teams when unusual behavior is detected, such as increased error rates or latency.

For example, you might configure an alert rule for high request latency:

groups:
  - name: Example Alerts
    rules:
      - alert: HighRequestLatency
        expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le)) > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High Request Latency"
          description: "Request latency is above 0.5 seconds for more than 5 minutes."

7. Container Monitoring

When deploying using Docker, ensure that metrics and logs from the containerized service are collected effectively. Utilizing a monitoring solution like Grafana alongside Prometheus can visualize performance metrics over time.

Use the following instructions to build and run a Docker container:

# Part of the Dockerfile for Sourcegraph/Zoekt
FROM golang:1.22.2-alpine3.19 AS builder

# Copy and download dependencies
COPY go.mod go.sum ./
RUN go mod download

COPY . ./
ARG VERSION
RUN go install -ldflags "-X github.com/sourcegraph/zoekt.Version=$VERSION" ./cmd/...

FROM alpine:3.19 AS zoekt

# Install necessary packages
RUN apk add --no-cache git ca-certificates tini
COPY --from=builder /go/bin/* /usr/local/bin/

ENTRYPOINT ["/sbin/tini", "--"]

Conclusion

Monitoring the Sourcegraph/Zoekt project in production is critical for maintaining service reliability and performance. By leveraging Go’s instrumentation capabilities alongside monitoring tools like Prometheus, developers can gain valuable insights into system behavior and take proactive measures to address potential issues.

Always ensure to refine monitoring setups according to the specific needs and loads of your service.