Self-Correcting RAG Using LLM-Based Retrieval Verification

Abstract:

Retrieval-Augmented Generation (RAG) reduces hallucination by grounding large language model outputs in externally retrieved evidence, but the standard single-pass pipeline has no mechanism to detect or correct a failed retrieval: when retrieved passages are irrelevant, insufficient, or noisy, the generator nonetheless conditions on them and produces a fluent but ungrounded answer. Prior work has addressed pieces of this problem in isolation, verifying retrieval adequacy, verifying output faithfulness, or adapting retrieval strategy to query complexity, but no widely reported architecture closes the loop at both the retrieval and generation stages within a single self-correcting pipeline. This paper proposes such a pipeline: an LLM-based retrieval verifier grades the relevance and sufficiency of retrieved evidence before generation, triggering corrective query reformulation and web-search fallback when retrieval is judged inadequate, while a second LLM-based verifier checks generated claims for attribution to the approved evidence after generation, triggering targeted regeneration of any claim it flags as unsupported. Drawing on established retrieval-augmented generation and RAG-evaluation literature, the study defines seven evaluation criteria — Answer Accuracy, Attribution Precision, Answer Recall, F1 Score, Hallucination Rate, retrieval-verifier AUC-ROC, and a combined Self-Correcting RAG Reliability Score — and applies them to an illustrative case study of an enterprise knowledge-assistant deployment with uneven corpus coverage. A five-configuration ablation study, spanning a naive single-pass RAG baseline through a deliberately over-engineered pipeline, is used to identify the point of diminishing returns, and the recommended configuration is benchmarked directly against the naive baseline, including performance under a retrieval-failure stress test of sparsely covered queries. The illustrative results indicate substantially higher recall, F1 Score, and a markedly lower hallucination rate for the proposed self-correcting pipeline relative to the baseline, with the largest relative improvement occurring specifically under sparse retrieval coverage. The findings position dual-stage, LLM-based self-correction as a foundational deployment practice for organizations operationalizing RAG where retrieval coverage cannot be guaranteed to be uniform or complete.