September 3, 2026 · 7 min read
Local AI for Your Business: How a GDPR-Compliant RAG System Works
"We'd like to use AI, but our contracts can't go into the cloud." I hear this sentence in almost every first conversation. The answer isn't to do without, but to use a different architecture: a RAG system that runs in-house, searches your documents and backs every answer with its source. This article explains how it works without jargon – and where the limits are.
What RAG means – in one paragraph
RAG stands for Retrieval-Augmented Generation. In plain terms: search first, then write. Before the language model writes an answer, the system pulls the relevant passages from your document store and hands them to the model as its basis. The answer is therefore not built from whatever the model once learned on the internet, but from your contracts, manuals, quotes and minutes.
The crucial side effect: because the system knows which paragraphs the answer came from, it can cite them. Every statement links to its source. That is the difference between a tool you can trust and a chatbot that merely sounds plausible.
Why your documents stay in-house
A RAG system has three parts: an index holding your documents split into searchable sections, a search that finds the matching sections for each question, and the language model that turns them into an answer. All three can run on a server in your own network – or on a rented server in a German data centre if you don't want to operate hardware yourself.
None of it is trained into the model. The documents live in the index; the model stays as it is. Delete a document and it is gone from the index – a deletion you can also prove to the data protection authority.
Three use cases that have proven themselves with clients
- Drafting quotes: the system reads comparable past quotes, the service specification and the framework agreement and delivers a draft with line items and prices – including which item is based on which document.
- Answering contract and policy questions: "What notice period applies with supplier X?" or "What does our travel policy say about rental cars?" – answered in seconds, with the paragraph cited.
- Taking load off onboarding and support: new employees ask the system instead of a colleague; answers stay consistent because they come from the same documents.
What GDPR-compliant operation actually requires
- A register of which document types may enter the index – HR files usually may not.
- Access rights that mirror the file system: anyone who can't open a file gets no answer from it either.
- Logging of queries and sources so results can be traced later.
- Human approval for anything that leaves the company – a quote goes out only after review.
- With a rented server: a data processing agreement and a location within the EU.
Hardware and a realistic timeline
For document search and text drafts, a server with a capable graphics card is usually enough for a small company. Alternatively a rented GPU server in a German data centre if you don't want hardware on site. The software itself is open source and needs no licence – I size both by data volume and number of users.
A first productive use case typically stands after three to five weeks: one week to review documents and settle access rules, two weeks to build and test with real questions from daily work, then a team briefing. Only once that one case works does the next one follow.
Local AI is not a compromise compared to the cloud – for many companies it is the only variant that can be justified at all with contracts, client or patient data. If you want to know whether your document base is suitable: I'll look at two or three typical questions from your daily work and tell you honestly whether a RAG system would answer them.
Using AI in your business?
I'll show you on a real use case what local AI can do for you.
Related service: Local AI & RAG Systems