LLMs coupled with OCR systems now make it possible to automate the processing of PDF or image documents: summaries, structured data extraction, chatbot feeding… But these models hallucinate. And for critical tasks where errors are not tolerated, a human must verify each result.
The problem: this verification is often as long as doing the work oneself. Faced with a 50-page document, the verifier must find the relevant information before being able to validate the LLM's response, a task the model has already done on its own, without leaving an exploitable trace.
The solution described here is based on a simple observation: the LLM knows where it found the information, you just need to ask it to show it. By exploiting the location metadata provided by the OCR, it is possible to highlight directly on the source document the text segment that justifies each element of the response. This is what is called a visual proof.
Proofs, yes, but at what level?
The idea of providing proofs to the user is not new. Connected chatbots (ChatGPT, Perplexity, Grok…) display clickable links to their sources. But these proofs remain “high-level”: to verify, you still have to read the source, which can be time-consuming.
The Google search engine offers a better model. For certain queries, it doesn't just list links: it displays the precise excerpt from the page that contains the answer, highlighting the relevant passage. It is this level of granularity that the system presented here transposes to document processing by OCR + LLM.
Example of proof in a connected chatbot (ChatGPT)
Example of visual proof in a Google search
User experience: click to verify
The interface is divided into two panes. On the left, the result of the LLM inference (summary, structured data, chatbot response…). On the right, the source document. When the user clicks on an element of the result, the right pane automatically displays the relevant page with the pertinent text segment highlighted.
The image at the top of the article illustrates this user interface.
What the OCR provides
The system relies on AWS Textract as an OCR engine (other tools provide equivalent metadata). For each detected element, Textract returns a block containing, among other things, the raw extracted text (Text), the position on the page in the form of a bounding box (BoundingBox), the block type (PAGE, LINE, WORD, TABLE, CELL, IMAGE…) and parent-child relationships between blocks (Relationships).
{
"BlockType": "LINE",
"Confidence": 97.36617279052734,
"Text": "MEUBLES à PIZZA",
"Geometry": {
"BoundingBox": {
"Width": 0.2342921942472458,
"Height": 0.02162465639412403,
"Left": 0.7166793942451477,
"Top": 0.028492335230112076
},
"Polygon": [
{ "X": 0.7167330980300903, "Y": 0.02968931570649147 },
{ "X": 0.9509715437889099, "Y": 0.028492335230112076 },
{ "X": 0.9509004354476929, "Y": 0.04892276972532272 },
{ "X": 0.7166793942451477, "Y": 0.05011698976159096 }
]
},
"Id": "63422f23-5ac4-4418-b002-9776ed79301b",
"Relationships": [
{
"Type": "CHILD",
"Ids": [
"9882688f-4d80-4835-87e2-3f1486295e1f",
"29ca488f-2fec-4464-9c7d-2c4b616e9a4d",
"f9c31d27-1fdc-460c-9fdd-5b8ee69d3e46"
]
}
]
}
It is these bounding boxes that allow knowing which pixels to highlight, and the hierarchical structure that allows controlling the granularity level of the proof (word, line, table…).
The prompt engineering challenge: staying token-lean
In a classic OCR + LLM pipeline, only the raw text is sent to the model. All other metadata is used only for formatting (tables in markdown, for example). Sending the entire Textract JSON to the LLM would be a disastrous naive approach: approximately 100,000 tokens per page, compared to 1,000 tokens for classic parsing.
The trick is to add metadata only at a high level of granularity. Specifically, only blocks of type line and table are numbered and tagged in the prompt via tags <LINE {id}> and <TABLE {id}>. Finer granularity blocks (words, cells) do not receive explicit metadata: the LLM can reference them via their parent and their textual value.
This choice of granularity is not arbitrary. Given the structure of Textract blocks, each piece of evidence can take four forms: a text block (approximately one word), a line, a table, or a table cell. For each task, the LLM can retrieve several pieces of evidence of different natures - for example, one line and three table cells.
Reconciling the LLM output with OCR blocks
As not all metadata is included in the prompt, the LLM may reference blocks that do not exist as such in the OCR output. Typical example: the OCR extracted 6 x 7 as three distinct blocks (",")6, x, 7), but the LLM references them as a single block. A deterministic post-processing step resolves this discrepancy by identifying, through textual similarity (fuzzratio), the OCR blocks whose combination best matches the LLM's reference.
The response structure: duplicate value and proof
For the system to work, the LLM's response must contain both the result and the associated location metadata. In practice, this means using structured outputs (JSON) by duplicating each field into a value and a proof.
Instead of a classic extraction:
{
"temperature": "4°C",
"power (kW)": 42,
"description": "SOME VERY LONG TEXT"
}
The response takes this form:
{
"temperature": {
"value": "4°C",
"proof": [{ "type": "WORD", "line": 2, "value": "4°C" }]
},
"power (kW)": {
"value": 42,
"proof": [{ "type": "WORD", "line": 5, "value": "6 x 7000 Watts" }]
},
"description": {
"value": "SOME VERY LONG TEXT",
"proof": [
{ "type": "LINE", "id": 19 },
{ "type": "LINE", "id": 20 }
]
}
}
Display: annotate the source image
Once the location metadata has been traced back to the Textract blocks, the system has the coordinates (between 0 and 1) of the four corners of each bounding box on the document.
Images are annotated by the backend. The frontend thus has access to the corresponding annotated image for each extracted data point.
Technically, image annotation relies on the library sharp, and PDF processing on pdf-lib.
