Overview

Production Monitoring is a critical aspect of managing the GitLab CE instance, ensuring reliability and performance in production environments. This document outlines the step-by-step processes and techniques used for monitoring the GitLab CE project effectively.

Monitoring Setup

Instrumentation

Instrumentation involves embedding monitoring capabilities directly into the GitLab codebase.

For monitoring purposes, GitLab uses Prometheus for metrics collection and Grafana for visualization. To instrument your instance, ensure that the prometheus service is enabled in your GitLab configuration:

monitoring:
  enabled: true
  prometheus:
    enable: true

Configuration

To properly configure monitoring, the following steps are essential:

1. Prometheus Configuration:

   Customize your Prometheus settings in the gitlab.yml file, particularly focusing on scraping configurations.

   prometheus:
     enable: true
     listen_address: ':9090'
     log_level: 'info'
   

2. Grafana Setup:

   Ensure Grafana is set up to connect to your Prometheus instance.

   grafana:
     enabled: true
     grafana_host: 'localhost'
     grafana_port: '3000'
   

3. Monitoring Alerts:

   Set up alerts for critical metrics. This can be done via alerting rules in Prometheus.

   groups:
     - name: gitlab-alerts
       rules:
         - alert: HighErrorRate
           expr: sum(rate(http_requests_total{job="gitlab"}[5m])) by (status) > 0.5
           for: 10m
           labels:
             severity: high
           annotations:
             summary: "High Error Rate detected"
             description: "High error rate detected over the last 10 minutes."
   

Key Metrics to Monitor

Monitoring relevant metrics is essential for maintaining application performance. Some key metrics include:

• Response Time:

  Monitor HTTP response times to identify latency issues.

  histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
  

• Database Query Performance:

  Keep track of slow database queries.

  rate(gitlab_database_queries_total{label="slow"}[5m])
  

• Sidekiq Job Performance:

  Monitor the performance of Sidekiq job processing.

  rate(sidekiq_queue_latency_seconds[5m])
  

Visualizing Metrics

Visualization is a significant aspect of monitoring; GitLab integrates with Grafana for this purpose. The dashboards should cover important aspects like:

• Request Metrics
• Database Latency
• Background Job Processing Times

Dashboard Example

To create Grafana dashboards, users can leverage pre-defined templates or establish custom queries.

Example of a panel configuration for response time:

{
  "title": "HTTP Response Time",
  "type": "graph",
  "target": [
    {
      "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))",
      "legendFormat": "{{status}}",
      "refId": "A"
    }
  ],
  "options": {
    "legend": {
      "show": true
    }
  }
}

Logging

In addition to metric monitoring, logging is critical in observing application behavior. GitLab uses a structured logging approach that includes handling for various environments and log levels.

Set up the logging in the configuration:

logging:
  log_level: 'info'
  log_format: 'json'

Conclusion

Effectively setting up production monitoring in GitLab CE involves careful configuration of metrics collection, alerting, and visualization. By leveraging tools like Prometheus and Grafana, along with key performance indicators and logging, teams can maintain the robustness of their GitLab installation.

Source

The information contained in this document is derived from the GitLab repository. More details can be found in the gitlab-ce codebase and documentation.