
Language model quality starts long before the first prompt — with how the documents it runs on are prepared. Here are the main steps of our process.

## 1. Corpus collection and cleaning

We start by gathering the source documents and bringing them to a single form: text extraction from PDF/DOCX/HTML, removal of boilerplate junk (headers/footers, navigation, duplicates), normalization of encoding and markup.

## 2. Structuring for RAG

Next, documents are structured for retrieval-augmented generation:

- **Chunking** — splitting into meaningful fragments, not a fixed number of tokens "blindly".
- **Metadata** — source, section, date, document type: this is what retrieval is later filtered and ranked by.
- **Retrieval verification** — a sample run of queries to make sure the needed fragments are actually found.

## 3. Formats for fine-tuning

If the data is being prepared for fine-tuning, we convert it to the target format (instruction/response, dialogs), validate examples for consistency and split the dataset into leak-free train/val/test.

## 4. Quality control

The final step is an audit: deduplication, completeness metrics, selective manual review. Only then does the data go to the client along with process documentation.

Got a corpus that needs tidying up? Write to us: [info@ai-engineering.space](mailto:info@ai-engineering.space).



When a RAG system answers off-target, the cause is almost never the model — it is the data. Here are the four problems we encounter most often.

## Bad chunking

Splitting "500 tokens with overlap" breaks the document's logic: a table ends up across three chunks, and its heading in a fourth. The fix is structural chunking that respects document markup.

## Missing metadata

Without document source, type and date you can neither filter retrieval results nor tell the model what to trust. Metadata is half of retrieval quality.

## Duplicates and junk in the corpus

The same document in five copies does not make retrieval more accurate — it only clogs the index. Deduplication before indexing is mandatory.

## No retrieval verification

The index is built, but nobody checked that typical queries actually find the right fragments. Running a control set of queries is a mandatory step before launch.

All of this is a data preparation problem, not a model architecture one. If your RAG is hallucinating, start with a corpus audit: [info@ai-engineering.space](mailto:info@ai-engineering.space).


How We Prepare Documents for LLMs

Why RAG Does Not Work: Typical Data Problems