In brief: Generative AI becomes risky as soon as it accesses corporate knowledge. Five risks occur regularly: lost access permissions, prompt injection, contradictory sources, answers without provenance, and data leaving your sphere of control. They can be mitigated with governance in two areas. The architecture must include permission checks prior to generation, evidence for every statement, and validation before every output. The organizational side requires source ownership, training, and documented data flows.

Why the risk arises with integration

A language model without access to your systems knows nothing confidential and consequently cannot disclose any of it. That changes with the very first connector. As soon as SharePoint, Confluence, the CRM, or document management systems are connected, any answer can draw on this content.

That is precisely why they are integrated. Only with company knowledge can a chatbot answer questions about products, contracts, and processes. Yet that same integration gives rise to the risks addressed in this article. The 2025 edition of the OWASP Top 10 for LLM Applications ranks Sensitive Information Disclosure in second place and has newly added Vector and Embedding Weaknesses as a standalone entry. Both directly affect systems that retrieve corporate knowledge.

Five risks

1. Access permissions are lost in the index

In SharePoint, a sales assistant is not permitted to open the compensation policy. In the search index of a simple RAG system, that same policy sits as a text chunk right next to the product catalog. If permissions are not retained during indexing and checked with every query, semantic similarity alone determines what ends up in an answer. Why classic RAG breaks down structurally at this point is described in our article on the Limits of Standard RAG in Customer Service.

2. Prompt injection via inputs and documents

Users can attempt to break a model out of its constraints using cleverly phrased inputs. A publicly documented case involves a Chevrolet dealership in Watsonville, California, whose chatbot in December 2023 was tricked into agreeing to sell a new vehicle to a user for one dollar as a "legally binding offer." When enterprise knowledge is integrated, a second vector emerges: instructions embedded within a document that are ingested during retrieval. OWASP ranks Prompt Injection at number one.

3. Contradictory and outdated sources

Corporate knowledge is rarely consistent. The same data sheet exists in three versions across two drives, and an expired price list remains accessible. A head of service described it in an initial consultation like this: "I don't know where which information is stored, or where it contradicts itself." A model that pulls from all sources absorbs these contradictions and phrases them convincingly.

4. Answers without provenance

If an answer provides no supporting evidence, it is impossible to determine in hindsight whether it originated from an approved document or was hallucinated by the model. This is a liability issue. In May 2026, the Higher Regional Court of Hamm (OLG Hamm) ruled that a company is liable for false statements made by its AI chatbot. The judgment is not yet legally binding. In the event of a dispute, being able to prove what a response was based on puts you in a significantly stronger position. The legal situation is described in detail in the article Are AI Chatbots Legally Permissible?

5. Data leaves your sphere of control

Many solutions send every query along with the retrieved context to an external model provider's API. Consequently, the very content that was integrated leaves the enterprise with every single request. It must be clarified where data is processed, whether queries are used for training, and which sub-processors are involved.

Governance in architecture

Effective controls reside within the system, outside of the conversation. Rules defined in the system prompt lose impact as conversation length increases and can be deliberately bypassed via prompt injection. Six controls therefore belong in the architecture:

1. Permission checks before generation. Every piece of information undergoes an authorization check before it reaches the language model. What the user is not permitted to see never makes it into the response.

2. Approved sources with versioning. Only verified sources are integrated, each with a version and an assigned owner. Outdated versions are archived.

3. Evidence for every statement. Every answer is traceable back to its source, document version, or rule.

4. Validation before every output. A dedicated validation step checks the response against the source before it is delivered.

5. Switchable generation. In critical factual scenarios, generative answers can be disabled. The bot then outputs approved standard texts or hands over the conversation.

6. Human handover and audit logging. If evidence is missing, an employee takes over with the complete interaction history. Every step is recorded in the audit log.

How such a boundary is defined for specific question types is described in the article What a Chatbot Should Not Answer.

Governance in the organization

Technology defines what the system is allowed to do. The organization defines who is accountable for it.

• Source ownership. For every connected source, an individual is assigned to approve content and ensure it remains up to date.

• Training. Anyone working with the system must understand what it can achieve and where its limits lie. The EU AI Act addresses AI literacy in Article 4.

• Transparency. Under Article 50 of the EU AI Act, users must be able to recognize that they are interacting with an AI system.

• Clear roles. As the operator, you are responsible for the purpose and deployment of the system, while the vendor is responsible for the technology. This includes a Data Processing Agreement (DPA) under Article 28 GDPR, a Data Protection Impact Assessment (DPIA) where required, and defined retention periods.

• An approved solution. Where none exists, employees query public chatbots about internal policies and receive answers devoid of context from your actual documents. Governance therefore also means providing a vetted, authorized tool.

How Mercury.ai implements this

• Hosted and operated in Germany. The platform runs exclusively on AWS Frankfurt, with no data transfer to third countries. The sole sub-processor is Amazon Web Services EMEA SARL, Germany Branch. Customer-managed keys are optionally available.

• No external model calls. Mercury.ai uses licensed, self-hosted Mistral models. Requests are never routed to an external model API, and customer data is not used for model training.

• Tenant isolation per customer. Every customer receives dedicated bot containers per environment, with no shared instances and no shared data.

• Controlled responses. Mercury Intelligence structures content within a knowledge graph, verifies it for consistency, and versions it. Permission checks are executed prior to generation, and a fact-check precedes every output. Every answer cites its source, significantly reducing the risk of hallucinations.

• Access control and operational security. Role-based access control and multi-tenant isolation, two-factor authentication, SCIM, comprehensive audit logging, regular penetration testing, and protection against prompt injection and knowledge base manipulation.

• Standards. Mercury.ai aligns with ISO 27001; the utilized data centers are ISO 27001 certified.

This approach is documented in our Security Paper on the EU AI Act and AI Safety.

Seven questions to ask before integration

• Are source system permissions validated with every query, or only during indexing?

• Who approves a source, and who is responsible for keeping it up to date?

• How does the system detect contradictory versions of the same document?

• Does every answer cite its source, and can this be verified retrospectively?

• Where are queries and retrieved contents sent, and are they used for training?

• How is the system protected against prompt injection via user inputs and documents?

• What happens when no supporting evidence is available?

Frequently asked questions

Is it fundamentally risky to use generative AI with corporate data?

The risk depends on the architecture. With permission checks prior to generation, approved sources, mandatory citations, and data processing within your own jurisdiction, it can be reduced to a manageable level. The key considerations regarding ChatGPT are detailed in the article Is ChatGPT GDPR-Compliant in Customer Service?

Is it enough to maintain permissions in the source system?

Only if the AI system inherits these permissions and enforces them with every query. Many RAG solutions store content in a unified index that strips away original access rights.

What is prompt injection?

An attack where inputs or document contents are crafted to make the model ignore its instructions. This can lead to unauthorized commitments or the disclosure of internal information. Effective defenses rely on controls external to the model, such as validation before every output.

Is a chatbot powered by corporate knowledge a high-risk system under the EU AI Act?

A service or knowledge chatbot typically does not fall into high-risk categories. It is primarily subject to transparency obligations. The decisive factor is the specific use case, for which the operator remains responsible.

Do we need a dedicated AI policy?

A concise policy outlining which tools are approved, what data may be processed, and who owns knowledge sources is highly advisable. It works best when paired with an officially approved tool that genuinely resolves employees' queries.

How do you prevent confidential documents from surfacing in chat?

By implementing permission checks that execute prior to generation, and through a deliberate selection of connected sources. What is not approved is never integrated.

Sources

• OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025, published on 18/11/2024. LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM08:2025 Vector and Embedding Weaknesses.

• Regulation (EU) 2024/1689 (EU AI Act), Article 4 (AI literacy) and Article 50 (Transparency obligations).

• OLG Hamm, judgment of 12/05/2026, Case No. 4 UKl 3/25: A company is liable for false statements made by its AI chatbot. Not yet legally binding, appeal on points of law admitted to the Federal Court of Justice (BGH).

• Chevrolet of Watsonville, California, December 2023: publicly reported case of a manipulated dealership chatbot.