BELIAEV.DEV/Senior Backend Engineer

Private AI

Private AI Document Search: How Local RAG Works

How to let people ask questions about internal documents with citations, using local embeddings and a local model, and where that design fails.

By Pavel Beliaev

Published

7 min read

Private AI document search is the ability to ask a question in ordinary language and get an answer grounded in internal files, without sending those files to a public AI service. The usual implementation is retrieval-augmented generation, RAG: find likely passages, then ask a model to answer using those passages.

RAG is not magic. It is a search pipeline with a language model bolted on the end. When it works, it is because the right text was retrieved. When it fails, it is usually because the right text was never retrieved, never parsed, or never allowed. If you want this built as a system rather than a notebook, that sits inside private AI consulting. This article is the mechanics and the failure modes.

The pipeline

Documents
    ↓
Parser
    ↓
Chunks
    ↓
Embeddings
    ↓
Vector Search
    ↓
Relevant Context
    ↓
LLM
    ↓
Answer + Sources

Every arrow is a place to lose information. Treat them as such.

Ingestion

Ingestion is the product. Chat is the demo.

You need a catalogue of sources: a fileshare, a Git repo of markdown, an export of Confluence, a bucket of PDFs, a dump of tickets. Each source has an owner, a refresh schedule, and an access list. If you cannot say who is allowed to see a drive, you cannot put it in the index.

Ingest incrementally. A nightly full re-embed of 80,000 files is how GPUs earn a reputation for being expensive. Watch mtime, etags, or a change feed. Delete from the index when a file is deleted or a permission is revoked. Tombstones matter more than new embeddings.

Record provenance: path, version, hash, ingested_at. When someone asks “why did it say that?”, you need to point at a file, not at a vibe.

Parsing

Models do not read PDFs. Parsers do.

  • Born-digital PDFs with a text layer: extract text, keep page numbers.
  • Scans: OCR first. Bad OCR in, confident nonsense out. Measure OCR quality on a sample before you promise Q&A.
  • Word and HTML: structure is useful (headings, tables). Flattening everything to a blob throws that away.
  • Spreadsheets: usually a poor RAG citizen. Tables want a database or a dedicated extractor, not a 500-token chunk of cell soup.
  • Slides: titles and notes matter; decorative text does not.

Keep page or section anchors. A citation that cannot open the source is decoration.

Parsing will fail on columns, headers, footers, and watermarks. Budget time for that, not for choosing a more fashionable embedding model.

Chunks

You cannot embed a 200-page policy as one vector and hope. You split.

Chunking is a trade-off:

  • too small: the passage has no meaning (“see section 4”);
  • too large: retrieval gets fuzzy and the prompt blows the context window;
  • naive splits in the middle of a table: the model sees half a grid.

Practical defaults that are allowed to be boring: 400–800 tokens, overlap of 50–100, split on headings where you have them. Store the parent document id so you can expand a hit to neighbouring chunks if the question needs more than one paragraph.

Do not chunk on a vibe. Take twenty real questions, look at what a human would highlight, and see whether your chunks contain that highlight.

Embeddings

An embedding model turns a chunk into a vector. The search then finds nearby vectors.

For private search, compute embeddings locally or in a private cloud you control. Using a hosted embedding API while proudly running a local LLM is a common way to leak the document text anyway. The same policy should cover every model, not only the chat model.

Embedding models are not interchangeable. A vector from model A is meaningless in an index built with model B. Pin the model name next to the index. When you change it, rebuild.

Quality: a solid open embedding model on domain text beats a famous general model that you cannot run locally. Test with your questions. There is no leaderboard for your policies.

Retrieval

Vector search is the default. It is not sufficient on its own.

Combine:

  • Keyword / BM25 for identifiers, names, statute numbers, error codes;
  • Vectors for paraphrases (“how do we treat parental leave” vs “family-related absence”);
  • Filters for source, date, language, and — this is the load-bearing one — permission.

Return the top k passages, then optionally rerank with a cross-encoder if you have the latency budget. k=8 with garbage is worse than k=4 with a filter.

If the corpus is small, start with keyword search plus a model. Teams skip to vectors because it sounds like the job. Sometimes grep-with-a-summary is the honest MVP.

Context and the LLM

The application builds a prompt:

  • system: answer only from the passages; if the passages are insufficient, say so; cite sources;
  • the retrieved passages, clearly delimited, with ids;
  • the user question.

Then a local generator produces an answer. The model is the last step, not the search.

Context windows are finite. Stuffing fifty chunks into a 32k window because you can is how you pay in latency and still miss the one paragraph that mattered. Rank, cut, and leave room for the question.

Different tasks can use different generators. A small model is enough when the answer is sitting in the passage. A larger one helps when you must compare two documents. That is an on-premise LLM deployment concern: one API, several model names.

Citations

If the UI cannot show the passage and a link to the file, you do not have document search. You have a chatbot that might be making it up.

Citations should be clicked. People will not audit markdown footnotes. Open the PDF on the page. Highlight the chunk if you can. When the model quotes something that is not in the chunk, that is a bug, not a style issue.

Refuse to answer without retrieval hits, unless you have a separately labelled “general knowledge” mode that is off by default for internal corpora.

Access permissions

This is the part demos skip and lawyers do not.

The index must know, per chunk, who may retrieve it. At query time, the filter is the user’s groups, not a hope. Building one giant index from a world-readable copy of the file server recreates the open share, with better grammar.

Practical approaches:

  • inherit ACLs from the source system if you can query them;
  • partition indexes by audience (legal, engineering, public-internal);
  • exclude drives you cannot classify.

A “CEO mode” that searches everything is an explicit product, with an explicit audit log, not a default.

Updating documents

Stale indexes are silent failures. A policy that was replaced in March will still be retrieved in September if you only ingested once.

Have:

  • a crawler or webhook;
  • a way to re-parse a single document;
  • a way to purge;
  • a visible “last indexed” on the source in the UI.

Versioned documents should retrieve the current version unless the user asks for history. Citing a superseded SOP as if it were in force is worse than a keyword search that misses.

Hallucination control

The model will still invent if you let it. Mitigations that actually move the needle:

  • retrieve first, generate second, always;
  • instruct it to quote or to point at passage ids;
  • answer “I don’t know” as a success state, not a failure;
  • keep temperature low for this task;
  • evaluate with questions whose answers you already know.

You will not eliminate hallucinations. You will make unsupported claims rarer and easier to catch. Anyone who promises zero is selling.

Evaluation

A demo on two PDFs is not evaluation.

Build a set of 30–100 questions from real people: the intern, the counsel, the on-call. For each, a human marks: right, incomplete, wrong, refused. Track retrieval hit rate separately from answer quality. If retrieval missed, do not blame the LLM.

Re-run the set when you change parser, chunker, embedder, or generator. That is your regression test. It is not glamorous. It is why the system stays honest.

Failure modes

Expect these, because they will happen:

  • The file was never ingested. The assistant cannot know about it. Surface “not in the index”.
  • The scan is unreadable. OCR produced alphabet soup. Retrieval returns it anyway.
  • The question needs calculation or judgement the documents do not contain. RAG will still try to sound sure.
  • The user does not have access. The correct behaviour is absence, not a leak.
  • The answer is in a table, a figure, or a footnote your parser dropped.
  • Names and numbers go wrong even when the passage is right. Show the passage.
  • Prompt injection in a document (“ignore previous instructions”). Treat retrieved text as hostile data, not as a system prompt.

None of these mean RAG is useless. They mean it is search infrastructure, and search infrastructure has operators.

Private document search is worth doing when people already drown in files, when the questions repeat, and when the corpus can be parsed. It is not a replacement for a records system, a wiki with decent titles, or teaching people to put documents in the right place.

If you want a bounded corpus indexed on infrastructure you control, with a question UI that shows sources, that is a standard pilot on the private AI deployment page. Start with one share and fifty questions, not with “all company knowledge”.

Related reading