pingcap / autoflow

Engine

The Engine is responsible for managing the interaction between the user and the RAG system. It provides functionalities for:

  • Input Processing: Handling user input and converting it into a format suitable for processing by the RAG system.
  • RAG Interaction: Sending the processed input to the RAG system and retrieving the generated responses.
  • Output Formatting: Formatting the RAG responses into a human-readable format and displaying them to the user.

Input Processing

The Engine receives user input in the form of text. It then performs the following steps to process the input:

  1. Tokenization: The input is tokenized into individual words or subwords. This step is crucial for the RAG system to understand the meaning of the input.
  2. Encoding: The tokens are encoded into numerical representations that can be processed by the RAG system.
  3. Preprocessing: The encoded input is preprocessed to remove irrelevant information or noise. This step ensures that the RAG system focuses on the essential parts of the input.

RAG Interaction

The Engine interacts with the RAG system to retrieve responses based on the processed input. The interaction involves:

  1. Sending Query: The processed input is sent to the RAG system as a query.
  2. Retrieval: The RAG system retrieves relevant documents or information from its knowledge base.
  3. Generation: The RAG system generates a response based on the retrieved information and the user’s query.

Output Formatting

The Engine receives the RAG response in a raw format. It then formats the response into a human-readable format, which can be displayed to the user. This involves:

  1. Decoding: The response is decoded from its numerical representation into text.
  2. Postprocessing: The decoded response is postprocessed to improve readability and remove any technical jargon.
  3. Display: The formatted response is displayed to the user in an interactive chat interface.

Examples

Input Processing

# Example input
          input_text = "What is the capital of France?"
          
          # Tokenization
          tokens = tokenizer.tokenize(input_text)
          
          # Encoding
          encoded_input = tokenizer.encode(tokens)
          
          # Preprocessing
          processed_input = preprocess(encoded_input)

RAG Interaction

# Send query to RAG system
          response = rag_system.query(processed_input)
          
          # Retrieve response
          response_text = response.get("text")

Output Formatting

# Decode response
          decoded_response = tokenizer.decode(response_text)
          
          # Postprocess response
          formatted_response = postprocess(decoded_response)
          
          # Display response
          display(formatted_response)

Code Files

The Engine code is implemented in the following files:

  • engine.py
  • input_processor.py
  • rag_interface.py
  • output_formatter.py

Dependencies

The Engine depends on the following libraries:

  • transformers
  • torch
  • rag

References