Your RAG Can Pass Individual Checks and Still Fail as a System

Your RAG Can Pass Individual Checks and Still Fail as a System

• AI & Machine Learning
• September 11, 2026

Enterprise RAG rarely gets a clean question, a perfect retrieval set, and one simple instruction. A real request looks more like this:

Summarize Q3 revenue by region. Use a markdown table. Cite the sources. If the data is incomplete, say so instead of estimating. If two reports conflict, surface both.

The model now has to satisfy several constraints at once: interpret the retrieved evidence, filter out noise, follow formatting rules, recognize missing data, handle conflicts, and know when to abstain.

A benchmark called EnterpriseRAG tests this exact problem using 983 expert-validated samples across six domains and 13 LLMs under three non-ideal retrieval conditions: retrieval noise, knowledge gaps, and factual conflicts.

The researchers measure instruction adherence using IAS (Instruction Adherence Score) in two ways:

• Loose IAS: the share of individual constraints satisfied.
• Strict IAS: a pass only when every constraint is satisfied in the same response.

The best result reached 83.8% Loose IAS, but only 26.8% Strict IAS.

That 57-point gap is the important part. High per-constraint performance can hide poor holistic compliance. A system may usually follow the requested format, cite evidence correctly, or satisfy individual instructions. But production consumes the final response, where all of those requirements have to hold at once.

The harder failures appear in what the authors call Knowledge Interaction Protocols: deciding whether the retrieved evidence is sufficient, refusing when it is not, and recognizing conflicts between sources. On queries with missing evidence, the best rejection accuracy was only 42.7%, while one model answered 93.4% of unanswerable queries instead of rejecting them. With conflicting evidence, even the strongest model identified the conflict in only 44.3% of cases.

That suggests a useful distinction for production RAG evaluation: per-metric and per-constraint tests are good for diagnosis. They are not enough for release criteria.

A production system also needs composite evaluations where grounding, instruction adherence, refusal behavior, conflict handling, and formatting all have to succeed in the same run under imperfect context.

The question is not only whether each part works in isolation. It is whether the system still behaves correctly when those requirements collide.

At DataObrii, we evaluate RAG systems at both levels: individual failure modes for debugging, and end-to-end behavior under noisy, incomplete, and conflicting context before deployment.

Source: “EnterpriseRAG: Benchmarking LLM Instruction Adherence and Robustness under Non-Ideal Enterprise Retrieval,” arXiv, August 2026.