In short: Generative AI becomes risky as soon as it accesses corporate knowledge. Five risks occur regularly: lost access rights, prompt injection, contradictory sources, answers without provenance, and data leaving your sphere of control. They can be mitigated through governance in two areas. The architecture must include pre-generation permission checks, source verification for every statement, and pre-output validation. The organization must encompass source ownership, training, and documented data flows.
Why the Risk Arises with Integration
A language model without access to your systems knows nothing confidential and consequently cannot disclose any of it. That changes with the very first connector. As soon as SharePoint, Confluence, the CRM, or document management systems are connected, any response can draw upon this content.
That is precisely why integration happens in the first place. Only with corporate knowledge can a chatbot answer questions about products, contracts, and processes. With that same integration, however, arise the risks discussed in this article. The 2025 edition of the OWASP Top 10 for LLM Applications ranks sensitive information disclosure in second place and has introduced weaknesses in vector and embedding stores as a new dedicated entry. Both directly affect systems that retrieve enterprise knowledge.
Five Risks
1. Access Rights Are Lost in the Index
In SharePoint, a sales assistant is not permitted to open the compensation policy. In the search index of a simple RAG system, that same policy sits as a text chunk right next to the product catalog. If permissions are not carried over during indexing and verified with every query, semantic similarity alone determines what ends up in a response. Why classic RAG structurally breaks down at this point is described in our article on the limits of standard RAG in customer service.
2. Prompt Injection via Inputs and Documents
Users can attempt to break a model out of its guardrails using cleverly crafted prompts. A publicly documented case is that of a Chevrolet dealership in Watsonville, California, whose chatbot was coaxed in December 2023 into offering a user a new vehicle for one dollar as a "legally binding offer." With integrated knowledge, a second attack vector emerges: instructions embedded within a document that are ingested during retrieval. OWASP lists prompt injection as its number one risk.
3. Contradictory and Outdated Sources
Enterprise knowledge is rarely consistent. The same data sheet exists in three versions across two network drives; an expired price list is still accessible. A Head of Customer Service described it during an initial consultation: "I don't know where which information is stored, or where it contradicts itself." A model that draws from all sources will adopt these contradictions and articulate them convincingly.
4. Responses Without Provenance
If a response provides no supporting reference, it is impossible to determine in hindsight whether it originated from an approved document or was hallucinated by the model. This is a liability issue. The Higher Regional Court (OLG) of Hamm ruled in May 2026 that a company is liable for false statements made by its AI chatbot. The ruling is not yet legally binding. Anyone who can substantiate the basis of an answer in a dispute is in a significantly better position. The legal situation is detailed in the article Are AI Chatbots Legally Permissible?
5. Data Leaves Your Sphere of Control
Many solutions send every query along with the retrieved context to the API of an external model provider. Consequently, the very enterprise content that was connected leaves the company with every request. It is critical to clarify where data is processed, whether queries are used for model training, and which sub-processors are involved.
Governance in Architecture
Effective controls reside within the system, outside the conversational flow. Rules placed within system prompts lose effectiveness as conversation length increases and can be deliberately bypassed via prompt injection. Therefore, six controls belong directly in the architecture:
Permission checks before generation. Every piece of information undergoes authorization verification before reaching the language model. What the user is not permitted to see never enters the response.
Approved sources with versioning. Only verified sources are connected, each with an assigned version and responsible owner. Outdated versions are archived.
Source attribution for every statement. Every answer is traceable back to its source, document version, or rule.
Pre-output validation. A dedicated validation step cross-references the answer with the source before it is delivered to the user.
Deactivatable generation. In high-stakes or critical factual scenarios, generative answers can be disabled. The bot then delivers approved static copy or initiates a handoff.
Human handoff and logging. If supporting evidence is missing, a human agent takes over with the complete interaction history. Every step is captured in the audit log.
How to define such boundaries for specific query types is explained in the article What a Chatbot Should Not Answer.
Governance in the Organization
Technology governs what the system is permitted to do. The organization governs who is accountable for it.
Source ownership. Every connected source must have an assigned owner who approves content and ensures it remains up to date.
Training. Anyone working with the system must understand what it can deliver and where its boundaries lie. The EU AI Act addresses AI literacy in Article 4.
Transparency. Under Article 50 of the EU AI Act, users must be informed that they are interacting with an AI system.
Clear roles. As the operator, you are responsible for the purpose and deployment of the system; the vendor is responsible for the technology. This requires a Data Processing Agreement (DPA) under Article 28 GDPR, a Data Protection Impact Assessment (DPIA) where necessary, and defined retention/deletion periods.
An approved solution. Where none exists, employees will turn to public chatbots with internal queries, receiving answers devoid of any reference to your internal documents. Governance therefore also includes providing an enterprise-approved tool.
How Mercury.ai Implements This
Hosted in Germany. The platform runs exclusively on AWS Frankfurt, without data transfers to third countries. The sole sub-processor is Amazon Web Services EMEA SARL, German Branch. Customer-managed keys are optionally available.
No external model API calls. Mercury.ai uses licensed, self-hosted Mistral models. Queries are never sent to external model APIs, and customer data is never used for training.
Customer isolation. Every customer receives dedicated bot containers per environment—no shared instances, no shared data.
Controlled answers. Mercury Intelligence structures content within a knowledge graph, verifies it for consistency, and versions it. A permission check occurs prior to generation, followed by a factual validation check before output delivery. Every answer cites its source, drastically reducing the risk of hallucinations.
Access and operational security. Role-based access control and multi-tenant isolation, two-factor authentication, SCIM, comprehensive audit logging, regular penetration testing, and protection against prompt injection and knowledge-base tampering.
Standards. Mercury.ai aligns with ISO 27001; the utilized data centers are ISO 27001 certified.
The implementation approach is documented in our Security Paper on the EU AI Act and AI Safety.
Seven Questions to Ask Before Integration
Are source system permissions checked with every query, or only during indexing?
Who approves a source, and who is accountable for keeping it updated?
How does the system detect contradictory versions of the same document?
Does every answer cite its source, and can this be verified retrospectively?
Where are queries and retrieved data sent, and are they used for model training?
How is the system protected against prompt injection via user inputs and ingested documents?
What happens when no verifiable source is available?
Frequently Asked Questions
Is it fundamentally risky to use generative AI with corporate data?
The risk depends on the architecture. With permission checks prior to generation, approved sources, mandatory source attribution, and processing within your own jurisdiction, it can be reduced to a manageable level. The key considerations regarding ChatGPT are detailed in the article Is ChatGPT in Customer Service GDPR-Compliant?
Is it enough to manage permissions solely in the source system?
Only if the AI system inherits these permissions and verifies them with every single query. Many RAG solutions store content in a shared index that strips out the original access rights.
What is prompt injection?
An attack where input prompts or document contents are crafted to override the model's instructions and guardrails. This can lead to unauthorized commitments or the disclosure of confidential data. Effective defenses rely on controls external to the model, such as pre-output validation.
Is a chatbot with access to enterprise knowledge considered a high-risk AI system under the EU AI Act?
A service or knowledge chatbot typically does not fall into the high-risk categories. It is primarily subject to transparency obligations. The decisive factor is the specific intended purpose for which the deployer is responsible.
Do we need a dedicated AI policy?
A concise policy outlining which tools are approved, which data may be processed, and who owns source content is highly recommended. It works best in conjunction with an approved enterprise tool that reliably answers employees' actual questions.
How do you prevent confidential documents from appearing in the chat?
Through pre-generation permission checks and the deliberate curation of connected sources. What is not approved for access must not be connected.
Sources
OWASP GenAI Security Project, OWASP Top 10 for LLM Applications 2025, published Nov 18, 2024. LLM01:2025 Prompt Injection, LLM02:2025 Sensitive Information Disclosure, LLM08:2025 Vector and Embedding Weaknesses.
Regulation (EU) 2024/1689 (EU AI Act), Article 4 (AI literacy) and Article 50 (Transparency obligations).
OLG Hamm, judgment of May 12, 2026, Case No. 4 UKl 3/25: A company is liable for false statements made by its AI chatbot. Not yet legally final; appeal on points of law permitted to the Federal Court of Justice (BGH).
Chevrolet of Watsonville, California, December 2023: publicly reported incident of a manipulated dealership chatbot.
