Local AI & RAG Systems – GDPR-compliant

Local AI for businesses means that the language model, the search index and your documents run in a place you control: on a server in your network or on a dedicated server in a German data centre. The local LLM is an open-source model that I install there. Your questions and documents go to no US provider, and none of it is used to train a model.

This simplifies the data protection questions that usually take longest with cloud AI. If everything runs in-house, there is no external AI provider acting as a processor and no transfer to a third country. With a rented server in the EU, a data processing agreement with the data centre is added. You still need a legal basis for the processing in both cases; running locally does not remove it.

GDPR-compliant AI is therefore not a product you buy but an operation you set up properly: which data may go in, who may see what, how long anything is kept. I prepare these points technically and in writing. You make the legal assessment together with your data protection officer, and my work is no substitute for legal advice.

In October 2025 the Datenschutzkonferenz, the body of Germany's independent federal and state data protection authorities, published guidance on generative AI systems using the RAG method. Its finding: RAG can improve the accuracy and traceability of answers and in many cases allows on-premise operation, so personal data does not go to online operators of large language models. New questions arise all the same.

If you want to use AI in a GDPR-compliant way, four practical points follow from it. A RAG system needs a legal basis too, because your documents are processed and stored in the index. Access belongs in a concept of rights and roles, separated by department or purpose. Index entries can be deleted individually, so retention periods apply there as well. And answers are only as good as current, complete documents.

That is how I set up a RAG system for businesses: permissions taken from your file storage, a register of approved documents, deletion right down to the index and logging of which sources backed an answer. The guidance is also clear about what RAG does not solve, such as questions about how the language model itself was trained. So I check and document the model's suitability, as the regulators recommend.

There are two ways to run it. A server on your own premises fits if you already have a server room or rack and the data should not leave the building at all. A dedicated server in a German data centre fits if you don't want to maintain hardware or several sites need access. The software is the same in both cases, so moving later remains possible.

The graphics card is what matters. Search and index run on ordinary server hardware, while the local LLM writing the answers benefits greatly from a graphics card with enough video memory. How large the model and card need to be depends on the number of users, answer length and language. Smaller models are enough for many tasks, and we find that out in the pilot before any hardware is bought.

I recommend starting with exactly one use case and sizing the hardware to it. There are also cases where local AI does not pay off: when only a handful of documents are involved, a question is rarely asked, or the task is really about figures and reports that a database delivers better. I tell you that openly before any effort is spent.

An AI knowledge base built from your own documents helps most where knowledge sits in folders and is searched for again and again. At BSU-Holding the system turns contracts, service specifications and past quotes into draft quotes that only leave the company after approval. Contract questions, onboarding knowledge for new staff, draft replies to recurring emails and summaries of minutes work just as well.

AI can be wrong, with RAG too. It can miss a passage, connect two documents incorrectly or answer from an outdated version. RAG does not rule that out, but it makes errors checkable: every answer names its sources, and one click shows the paragraph. That is why a person stays in the approval loop for anything that goes out, and outdated documents are removed from the index.

Whether it works for you only shows in a pilot on real data with questions from your daily work. I work remotely or on site in Berlin & Brandenburg, so AI for businesses in Berlin also means one contact person who knows your network. Instead of AI consulting with slides, you get a working prototype and an honest measurement of whether it saves time.