GitLab Sidekiq and Background Jobs

Introduction

GitLab utilizes Sidekiq for processing background tasks. This document outlines how GitLab leverages Sidekiq, including:

• Job Queues: How jobs are queued and processed.
• Worker Processes: How worker processes handle queued jobs.
• Scheduling: How jobs are scheduled for execution at specific times.

Job Queues

Jobs are placed into queues to be processed asynchronously.

• Queues: Jobs are grouped into queues based on their type and priority. For example, default, high, low, cron, and scheduled are common queues.
• Job Creation: Jobs are created using the enqueue method of the Sidekiq::Client class.

# app/models/user.rb
          class User < ApplicationRecord
            def send_welcome_email
              # ...
              Sidekiq::Client.enqueue(WelcomeEmailWorker, self.id)
            end
          end
          

• Processing: Worker processes consume jobs from queues.

Worker Processes

• Workers: Workers are Ruby classes that implement the perform method, which defines how to process the job.

# app/workers/welcome_email_worker.rb
          class WelcomeEmailWorker
            include Sidekiq::Worker
          
            def perform(user_id)
              user = User.find(user_id)
              # ... send welcome email to user
            end
          end
          

• Worker Configuration: Worker processes are configured in the config/sidekiq.yml file. This file defines the number of worker threads, the retry strategy, and other settings.

# config/sidekiq.yml
          development:
            :concurrency: 25
            :queues: [:default, :high, :low, :cron, :scheduled]
            :redis:
              :url: redis://localhost:6379/0
              :namespace: sidekiq
            :pidfile: tmp/pids/sidekiq.pid
            :logfile: log/sidekiq.log
            :daemon: true
            :environment: development
          

• Worker Management: Workers are managed using the sidekiq command-line tool.

# Starting Sidekiq
          $ sidekiq -C config/sidekiq.yml
          
          # Stopping Sidekiq
          $ sidekiqctl stop
          

Scheduling

• Scheduling Jobs: Jobs can be scheduled for execution at specific times or intervals using the Sidekiq::Cron::Job class.

# app/jobs/daily_report_job.rb
          class DailyReportJob
            include Sidekiq::Job
            sidekiq_options queue: :cron
          
            def perform
              # Generate and send daily report
            end
          end
          
          # config/initializers/sidekiq_cron.rb
          require 'sidekiq/cron'
          
          Sidekiq::Cron::Job.create(name: 'Daily Report', cron: '0 0 * * *', class: 'DailyReportJob')
          

• Cron Expressions: Cron expressions define the schedule for jobs. For example, 0 0 * * * schedules a job to run at midnight every day.

Conclusion

By using Sidekiq, GitLab efficiently processes background tasks, freeing up resources for user-facing operations.

References

• Sidekiq Documentation
• GitLab Sidekiq Configuration