ShareLinkedInXFacebookEmail

Local RAG System

OtherVisit

Project Overview

Local RAG System is a context-aware document ingestion and semantic retrieval application built in Python. It converts uploaded documents into searchable vector embeddings and returns the most relevant text chunks for a user's query. The retrieved results can be used as context by an AI assistant or another downstream application.

The project provides both a Streamlit interface and a FastAPI service. It can store embeddings in PostgreSQL with pgvector or in a local JSON file, making the same retrieval workflow suitable for database-backed and lightweight local setups.

The Problem

Follow-up questions in a conversation are often incomplete when read on their own. A query such as “What does it cost?” may not contain enough information for vector search to retrieve the right document section. RAG applications also need a reliable way to parse different file formats, divide their content into useful chunks, prevent unnecessary re-indexing, and keep separate knowledge bases isolated.

My Solution

I built a modular retrieval pipeline that:

  1. Accepts TXT, CSV, PDF, and DOCX uploads.
  2. Extracts and chunks their content using configurable section, line, or word-budget strategies; CSV files are indexed row by row.
  3. Creates normalized 384-dimensional embeddings with intfloat/multilingual-e5-small, using the E5 passage: and query: prefixes.
  4. Stores and searches vectors through either PostgreSQL/pgvector or a local JSON/NumPy backend.
  5. Uses a local GGUF model through llama-cpp-python to turn conversational follow-up questions into standalone search queries.
  6. Filters retrieval by tenant and knowledge-base identifiers and returns ranked chunks with cosine-similarity scores and source metadata.

Key Features

  • Two interchangeable vector-storage backends: PostgreSQL with pgvector or local JSON with NumPy
  • Multilingual semantic retrieval using the E5 embedding model
  • Optional local query rewriting based on up to five previous questions
  • Graceful fallback to the original query when the query-rewriter model is unavailable
  • TXT, CSV, PDF, and DOCX ingestion
  • Configurable chunk size and splitting strategy
  • SHA-256 content hashes to skip unchanged chunks during upsert
  • Tenant-aware and knowledge-base-aware search filtering
  • Selectable top-k retrieval from the Streamlit interface and API
  • Knowledge-base statistics plus file-level and knowledge-base-level deletion
  • Search results with similarity score, source filename, metadata, and chunk ID
  • Separate timing for query rewriting, embedding generation, and vector search
  • FastAPI /search and /health endpoints
  • One launcher for the Streamlit UI and FastAPI service

How It Works

Document upload
      |
      v
Text extraction and chunking
      |
      v
E5 passage embeddings
      |
      v
PostgreSQL/pgvector or local JSON storage

User query + recent history
      |
      v
Optional local query rewriting
      |
      v
E5 query embedding
      |
      v
Tenant/knowledge-base filtered similarity search
      |
      v
Ranked source chunks for downstream RAG use

Technical Implementation

The PostgreSQL backend creates a dimension-specific vector table and an HNSW cosine index automatically. It uses pgvector's cosine-distance operator for ranked retrieval. The local backend stores the same records in JSON and calculates cosine similarity with NumPy; because the embeddings are normalized, the dot product provides the similarity score.

Each indexed chunk includes its tenant ID, knowledge-base name, source filename, record identifier, content hash, text, and embedding. When a chunk with the same ID and content hash already exists, the system skips re-embedding it.

The query rewriter runs locally from a GGUF model with a 2,048-token context window and a 256 MB inference cache. Rewriting preserves the query's original language and only runs when conversation history and the local model are available.

My Contributions

  • Designed the end-to-end ingestion and semantic retrieval workflow
  • Built the Streamlit pages for uploading, searching, managing data, and viewing API usage
  • Developed the FastAPI search interface and health endpoint
  • Implemented PDF, DOCX, CSV, and plain-text extraction and chunking
  • Integrated multilingual E5 embeddings with retrieval-specific prefixes
  • Implemented PostgreSQL/pgvector search with an HNSW index
  • Built the alternative JSON/NumPy vector store for local use
  • Added contextual query rewriting with a local GGUF model
  • Added tenant and knowledge-base filtering for data separation
  • Added content-hash-based update detection and knowledge-base management tools
  • Added component-level latency reporting and source-aware search results
  • Created a unified launcher for the UI and API processes

Technology Stack

  • Python
  • Streamlit
  • FastAPI and Uvicorn
  • Sentence Transformers
  • intfloat/multilingual-e5-small
  • llama.cpp via llama-cpp-python
  • PostgreSQL and pgvector
  • NumPy and JSON
  • PyPDF
  • python-dotenv

Outcome

The result is a working, locally deployable retrieval layer for RAG applications. Users can ingest documents, organize them by tenant and knowledge base, perform context-aware semantic searches, inspect ranked source chunks, and access retrieval through either a graphical interface or an HTTP API. The modular storage design also allows the project to move between a simple file-based setup and PostgreSQL without changing the ingestion or search experience.

Current Scope

This project provides the retrieval component of a RAG pipeline. It returns relevant document chunks but does not currently send that context to a separate generative model to produce a final answer. Retrieval-quality benchmark scores and production-scale performance results have not yet been measured, so no unverified metrics are claimed here.