stefanprodan / timoni

Troubleshooting and Debugging - stefanprodan/timoni

Troubleshooting and Debugging for Timoni project covers common issues and errors, debugging techniques, and tools. Timoni is a package manager for Kubernetes, inspired by Helm and powered by CUE. It aims to improve the UX of authoring Kubernetes configs.

Common Issues and Errors:

  1. Unable to find traces in Tempo: This issue can occur due to problems in ingestion or querying of data into Tempo. To diagnose and fix ingestion issues, enable debug logging of all traces received by the distributor and check the value of tempo_distributor_spans_received_total and tempo_ingester_traces_created_total metrics.
  2. Spans are being refused with “pusher failed to consume trace data”: This error can occur if the Grafana Agent is not sending traces to the backend. To query traces, check for error messages such as “Too many jobs in the queue”, “error using pageFinder”, or “response larger than the max”.

Debugging Techniques and Tools:

  1. Timoni mod lint: This command checks the syntax and structure of the Timoni module and returns any errors or warnings.
  2. timoni bundle lint: This command checks the syntax and structure of the Timoni bundle and returns any errors or warnings.
  3. distributor.log-received-traces: This flag enables debug logging of all traces received by the distributor and can help check if Tempo is receiving any traces at all.
  4. Verifying Image Signatures: Chainguard Images contain verifiable signatures and high-quality SBOMs, enabling users to confirm the origin of each image built and have a detailed list of everything that is packed within.

Debugging Techniques for Tempo:

  1. Checking Ingestion Issues: To diagnose issues in ingestion, add the distributor.log-received-traces flag to the distributor container, check the value of tempo_distributor_spans_received_total and tempo_ingester_traces_created_total metrics, and check if the application spans are reaching Tempo.
  2. Checking Querying Issues: To diagnose issues in querying, check for error messages such as “Too many jobs in the queue”, “error using pageFinder”, or “response larger than the max”.

Debugging Techniques for Bad Blocks:

  1. Fixing Bad Blocks: A backend block can be fixed if either the index or bloom-filter is corrupt/deleted. To fix such a block, download it onto a machine where you can run the tempo-cli, and run the gen index or gen bloom commands depending on which file is corrupt/deleted.

Sources: