Codebase

Navigate the Codebase using the tree view. Use the filters to highlight where various files are grouped to aid understanding.

Getting Started Commands

go get github.com/sourcegraph/zoekt/: Download and install the Zoekt package

go install github.com/sourcegraph/zoekt/cmd/zoekt-index: Install the Zoekt indexing command

$GOPATH/bin/zoekt-index .: Index the current directory

go install github.com/sourcegraph/zoekt/cmd/zoekt-git-index: Install the Zoekt Git indexing command

$GOPATH/bin/zoekt-git-index -branches master,stable-1.4 -prefix origin/ .: Index the Git repository with branches ‘master’ and ‘stable-1.4’

go install github.com/sourcegraph/zoekt/cmd/zoekt-{repo-index,mirror-gitiles}: Install the Zoekt repo indexing and mirror-gitiles commands

zoekt-mirror-gitiles -dest ~/repos/ https://gfiber.googlesource.com: Mirror Gitiles repositories to the specified directory

zoekt-repo-index ...: Index the specified repository with the given configuration

go install github.com/sourcegraph/zoekt/cmd/zoekt-webserver: Install the Zoekt webserver command

$GOPATH/bin/zoekt-webserver -listen :6070: Start the Zoekt webserver on port 6070

`curl –get \

    --url "http://localhost:6070/search" \
          
              --data-urlencode "q=ngram f:READ" \
          
              --data-urlencode "num=50" \

–data-urlencode “format=json”`: Search using the Zoekt JSON API

go install github.com/sourcegraph/zoekt/cmd/zoekt: Install the Zoekt command-line interface

$GOPATH/bin/zoekt 'ngram f:READ': Search for the term ‘ngram’ in files with the ‘READ’ file extension using the Zoekt CLI

Create a systemd unit file for the Zoekt webserver


          [Unit]
          
          Description=zoekt webserver
          
          [Service]
          
          ExecStart=/zoekt/bin/zoekt-webserver -index /zoekt/index -listen :443  --ssl_cert /zoekt/etc/cert.pem   --ssl_key /zoekt/etc/key.pem
          
          Restart=always
          
          [Install]
          
          WantedBy=default.target

Install and configure the Zoekt indexserver


          go install github.com/sourcegraph/zoekt/cmd/zoekt-indexserver
          
          cat << EOF > config.json
          
          [{"GithubUser": "username"},
          
          {"GithubOrg": "org"},
          
          {"GitilesURL": "https://gerrit.googlesource.com", "Name": "zoekt" }
          
          ]
          
          EOF
          
          $GOPATH/bin/zoekt-indexserver -mirror_config config.json

Install Universal ctags for improved ranking

go install github.com/universal-ctags/ctags

Refer to the ctags.md documentation for more information.

Project Structure

Below is a short description of the key directories of the project to aid you when understanding where key files are located.

.bazelignore, .dockerignore, .tool-versions, Dockerfile, Dockerfile.indexserver, Dockerfile.webserver, LICENSE, README.md, all.bash, api.go, api_proto.go, api_proto_test.go, api_test.go, bits.go, bits_test.go, btree.go, btree_test.go, contentprovider.go, contentprovider_test.go, eval.go, eval_test.go, flake.lock, flake.nix, gen-proto.sh, go.mod, go.sum, hititer.go, hititer_test.go, index_test.go, indexbuilder.go, indexdata.go, indexdata_test.go, indexfile_other.go, indexfile_unix.go, install-ctags-alpine.sh, limit.go, limit_test.go, marshal.go, marshal_test.go, matchiter.go, matchtree.go, matchtree_test.go, merge.go, merge_test.go, read.go, read_test.go, score.go, score_test.go, section.go, shell.nix, toc.go, tombstones.go, tombstones_test.go, tombstones_unix.go, tombstones_windows.go, write.go - These are various files in the repository, including configuration files, build scripts, and source code files.

.vscode/ - This directory contains Visual Studio Code specific configuration files for the project.

build/ - This directory contains the build files for the project.

build/testdata/ - This directory contains example files used for testing. It includes files in various programming languages such as C, Java, Kotlin, Python, Ruby, and Scala.

cmd/ - This directory contains the command-line interface (CLI) for Zoekt.

cmd/zoekt-archive-index/ - This directory contains the code for building and managing archive indexes.

cmd/zoekt-dynamic-indexserver/ - This directory contains the code for the dynamic index server.

cmd/zoekt-git-clone/ - This directory contains the code for cloning Git repositories.

cmd/zoekt-git-index/ - This directory contains the code for managing Git indexes.

cmd/zoekt-index/ - This directory contains the code for managing indexes in general.

cmd/zoekt-indexserver/ - This directory contains the code for the index server.

cmd/zoekt-merge-index/ - This directory contains the code for merging indexes.

cmd/zoekt-mirror-bitbucket-server/, cmd/zoekt-mirror-gerrit/, cmd/zoekt-mirror-github/, cmd/zoekt-mirror-gitiles/, cmd/zoekt-mirror-gitlab/ - These directories contain the code for mirroring code hosting platforms.

cmd/zoekt-repo-index/ - This directory contains the code for managing repository indexes.

cmd/zoekt-sourcegraph-indexserver/ - This directory contains the code for the Sourcegraph index server.

cmd/zoekt/ - This directory contains the main package for the CLI tool.

ctags/ - This directory contains the code for the Ctags parser.

debugserver/ - This directory contains the code for the debug server.

doc/ - This directory contains documentation for the project.

gitindex/ - This directory contains the code for managing Git indexes.

grpc/ - This directory contains the code for gRPC support.

internal/ - This directory contains internal packages used by the project.

json/ - This directory contains the code for handling JSON data.

query/ - This directory contains the code for query processing.

shards/ - This directory contains the code for managing shards.

testdata/ - This directory contains test data for the project.

trace/ - This directory contains the code for tracing.

web/ - This directory contains the code for the web server.

Entrypoints

Below are files we identified as entrypoints for to the codebase. This is where the application starts and a good place to start when learning.

cmd/zoekt-archive-index/main.go - The main entrypoint for the archive index binary, which indexes an archive.

cmd/zoekt-dynamic-indexserver/main.go - The main entrypoint for the dynamic index server, handling logging, indexing repositories, and empty directories.

cmd/zoekt-git-clone/main.go - The main entrypoint for the git clone binary, which fetches and clones repositories from various sources.

cmd/zoekt-git-index/main.go - The main entrypoint for the git index binary, which indexes a git repository.

cmd/zoekt-index/main.go - The main entrypoint for the index binary, which indexes a repository or an archive.

cmd/zoekt-indexserver/main.go - The main entrypoint for the index server, handling logging, periodic fetching, indexing pending repos, and deleting logs.

cmd/zoekt-merge-index/main.go - The main entrypoint for the merge index binary, which merges indexes from multiple shards.

cmd/zoekt-mirror-bitbucket-server/main.go - The main entrypoint for the bitbucket-server mirror binary, which fetches and clones repositories from a bitbucket server.

cmd/zoekt-mirror-gerrit/main.go - The main entrypoint for the gerrit mirror binary, which fetches and indexes repositories using the gerrit protocol.

cmd/zoekt-mirror-github/main.go - The main entrypoint for the github mirror binary, which fetches and clones repositories from a GitHub user or organization.

cmd/zoekt-mirror-gitiles/main.go - The main entrypoint for the gitiles mirror binary, which fetches and indexes repositories from a gitiles server.

cmd/zoekt-mirror-gitlab/main.go - The main entrypoint for the gitlab mirror binary, which fetches and indexes repositories using the gitlab API.

cmd/zoekt-repo-index/main.go - The main entrypoint for the repo-index binary, which indexes a repository-based repository using a manifest file.

cmd/zoekt-sourcegraph-indexserver/main.go - The main entrypoint for the sourcegraph index server, which periodically reindexes enabled repositories on Sourcegraph.

cmd/zoekt-test/main.go - The entrypoint for the zoekt-test binary, which compares the search engine results with raw substring search.

cmd/zoekt-webserver/main.go - The main entrypoint for the web server, which serves search results and handles various HTTP requests.

cmd/zoekt/main.go - The main entrypoint for the Zoekt command-line tool, providing functions for displaying matches, loading shards, and starting various profiling modes.

testdata/repo/main.go - Not an entrypoint, but a test data file for a repository.

testdata/repo2/main.go - Not an entrypoint, but a test data file for another repository.