Replacing GPT with a self-hosted open-source model: feedback on document extraction

In brief: actual migration of a document extraction stack fromAzure (Document Intelligence + GPT-4o) to an self-hosted open-source SLM on a single GPU. Result: cloud-equivalent quality, faster processing, and up to 20× cheaper at scale - without any sensitive data leaving the infrastructure. Relevant from a few thousand pages/month, under constraint of sovereignty or rate limits.

Document extraction is one of the generative AI tasks that can be easily handled with proprietary API models like GPT, Claude, and Gemini. These models are quick to integrate, performant, and simple to operate. However, their use quickly runs into a conjunction of several constraints: data sovereignty, the cost charged per token which increases with volume, rate limits which cap throughput, moderation which sometimes blocks legitimate requests, and vendor lock-in.

However, the performance of open-source SLMs (small language models) now matches that of large proprietary models for classic tasks (OCR, segmentation, structured extraction). At Galadrim, a French tech & AI agency, we carried out this migration for a client subject to strong sovereignty constraints. This article details the step-by-step feedback: model and GPU selection, putting into production, safeguards, and evaluation method to validate that no quality is lost.

The context: why a proprietary API was no longer viable

The project is a bespoke business application built for a medical expertise firm whose clients are health insurers in Switzerland. A user uploads a composite PDF file, often 50 to 500 pages, which gathers heterogeneous documents: reports, forms, letters, prescriptions, invoices. The application breaks down this PDF into distinct sub-documents, extracts the date and title, produces a summary, and selects the useful documents for processing.

Internally, this pipeline involves 7 stages :

page rendering as images,
OCR,
segmentation (defining boundaries between sub-documents),
block merging,
date extraction,
content extraction (title + summary + selection),
and deduplication.

The segmentation is the pivotal step. If we cut incorrectly, everything else is skewed: a report split into several pieces explodes the extracted dates and skews the selection. So, this is the step on which we first concentrated our tests.

The private data protection was at the heart of the project's development constraints due to sensitive data processed and regulated by the FADP Swiss (Federal Act on Data Protection), equivalent to European regulations. Several clients of the solution wished to go beyond the contractual guarantees provided by Azure in Switzerland in terms of data sovereignty.

To this regulatory constraint, were already added a cost billed per token which increases with volume, and a dependency on providers outside our control. The client therefore requested, upstream, an infrastructure 100% Swiss, with no non-European third parties involved. This is the trigger for the migration.

Choosing your SLM: multimodality and VRAM budget

Migrating to self-hosted GPU involves an infrastructure cost and operational complexity and mainly raises the following questions: what infrastructure to choose for inference, particularly GPU Random Access Memory (VRAM) and what model to choose for each task (number of parameters, quantification etc.)?

The LLM for extraction and synthesis tasks

The input for the segmentation step consists ofimages of pages. Therefore, a model is needed that is multimodal, with a large context window, that fits on a single GPU.

Model	Active Params	VRAM Q5	Multimodal	Choice
Llama 3.1 70B	70B	~50 GB	✗	✗no vision
Mistral Large 2 123B	123B	> 80 GB	✗	✗unfeasible single-GPU
Qwen2.5-VL 32B	32B	~22 GB	✓	▲correct but slow
Qwen3.6-35B-A3B (MoE)	3B active / 35B	~28 GB	✓	✓selected
Pixtral 12B	12B	~10 GB	✓	✗quality gap

Table 1 - Multimodal model choice (splitter): A single 48 Go GPU, vision required to read pages.

We chose a Mixture-of-Experts (Qwen3.6-35B-A3B): it only activates 3 billion parameters per generated token while possessing the knowledge of a 35 billion parameter model. We therefore obtain the throughput of a small model with the quality of a large one. For our task, it is on par with GPT-4o in segmentation and date extraction.

The OCR

GLM-OCR is OCR specialised (not general-purpose) and reads handwritten texts well. This is a critical point for forms where the useful information is often a handwritten note.

Model	Small text quality	Speed	Choice
Azure Document Intelligence	reference	1-2 s/p	✗cloud
Tesseract / PaddleOCR	degraded on stamps/handwriting	< 1 s/p	✗insufficient
Qwen2.5-VL OCR-mode	good	8-10 s/p	▲slow
GLM-OCR (THUDM)	≈ Azure DI	1-2 s/p	✓selected

Table 2 - OCR engine choice: Quality close to d'Azure, without the cloud.

Quantization : it's primarily a question of VRAM budget

Running two models on 48 GB is primarily a problem ofmemory footprint. We compared several levels of quantization and chose, for our case and our VRAM, Q5_K_M for the LLM and Q8_0 for the OCR: this is the point where quality remains indistinguishable from FP16 on our tasks while leaving enough headroom to run several concurrent LLM queries. The right level depends on the model, the task, and the GPU. Q4 degraded structured extraction and OCR of small numbers; FP16 simply wouldn't fit.

Figure 1 - The two models coexist on a single 48 Go GPU: ~38 Go allocated, ~10 Go kept in margin to absorb context peaks.

Two lessons learned from putting SLMs into production

Disable « thinking mode »: structure the task rather than letting the model think

By default, the model outputs several thousand reasoning tokens (…) before producing its JSON. On a task with structured output, this reasoning is wasted time: on a batch of 10 pages, we go from ~20 s to ~3 s by disabling it, a gain × 5 to × 10.

Beyond the flag: on structured tasks, quality comes less from free reasoning than from clearly defining and structuring the task in the system prompt. And when output quality is critical, implicit reasoning is replaced by an explicit and controlled mechanism, typically multiple passes in an adversarial LLM-as-a-judge network to strengthen and validate the prompt system. We keep reasoning only where it genuinely helps (here, deduplication).

Always test your guardrails: open-source models hallucinate

A self-hosted SLM does not have the safety net of a vendor. You need to build your own guardrails, and test them under real-world conditions to find the right solutions.

Guardrail	Trigger	Effect
OCR loop detection	Abnormally long output or phrase-level repetition	Save valid prefix, retry, fallback
Fallback per-page	OCR batch parsing fails	Re-OCR page by page
LLM Fallback on OCR loop	OCR loop + failed retry	Switch to LLM vision for this page
Exponential Retry	Erreur HTTP / timeout	Recovers from network transients or saturation
Boundary resolver	The splitter hesitates between 2 boundaries	Re-injects the bordering context to decide

Table 3 - Guardrails for the self-hosted pipeline: without a supplier safety net, everything is managed on the pipeline side.

In practice, these guardrails recover almost all incidents: on our batches, approximately 1% of pages enter an hallucination loop, and 100% are recovered via valid prefix backup + fallback. The objective is to detect and to have a systematic plan B.

Measure to decide: how we validated that we weren't losing anything

The number one risk of a migration like this is to silently degrade quality. So we built a harness evaluation that compares each output of the local model to a reference, on a set of annotated representative files (several hundred sub-documents).

What reference? The target is theextraction done upstream by GPT-4o (the historical production stack), consolidated by annotation. Important point: this reference is not an absolute truth. There remain business uncertainties and it happens that the LLM-as-a-judge considers that the segmentation or extraction of the local model are better than the reference. In other words, some of the measured 'discrepancies' are not regressions but legitimate disagreements, which mechanically caps raw scores downwards.

Separate the mechanical from the semantic. Everything that is mechanically comparable (boundaries, dates, selection, hallucinations) is calculated in pure Python, alignment by page overlap, F1, which makes these KPIs bit-by-bit reproducible between two runs. But some criteria cannot be reduced to an exact-match : "Dr Martin Medical Report" and "Initial Report - Jean MARTIN" refer to the same document. For these cases, a LLM-as-a-judge renders a structured verdict ok | partial | ko + a short reason. Only this layer depends on an LLM, so it remains independently auditable.

KPI	Definition	Target
Cut F1 (tol ±1)	Precision/recall of segmentation boundaries	≥ 90 %
Date exact rate	% of exact dates predicted	≥ 80 %
is_selected F1	Selection of useful items	≥ 90 %
Hallucination rate	% of predicted documents with no match	≤ 2 %
Title verdict	Title quality, judged by LLM	≥ 95 % ok

Table 4 - KPIs and business targets: separating the mechanical from the semantic.

Tip: the LLM-as-a-judge as an annotator when you don't have a golden set. Building a ground truth by hand is expensive. With current models (we use today Claude Code avec Opus 4.8, or Codex avec GPT-5.5), an LLM-as-a-judge makes a surprisingly reliable initial annotator: it pre-annotates, we review, and we iterate. It's an excellent way to bootstrap an evaluation set at a lower cost.

⚠️ For sensitive data, anonymise first. As soon as content is sent to an LLM-as-a-judge or an annotation model - a fortiori via an external API - it is necessary to anonymise upstream (redaction of identities, identifiers). Otherwise, you reintroduce through the back door exactly the data leak you were trying to eliminate by migrating on-premise. Hence the importance of an anonymised baseline set.

(Cross-cutting Note) To iterate quickly, it's a code agent that drives the evaluation chain : it launches the pipelines, triggers the judgment passes, aggregates the summaries, and regenerates a cross-run comparison - without manual intervention. A good example of using code agents to accelerate an R&D loop.

Results: quality, speed, cost

Quality delivered

KPI	Score (local)	Verdict
Cut F1 (tol ±1)	89,8 %	✓at target level
Date exact rate	81,3 %	✓
is_selected F1	85,1 %	✓
Hallucination rate	0,4 %	✓almost zero
Title verdict ok	68 % ok / 25 % partial / 6 % failed	✓

Table 5 - Local SLM results vs. targets: all indicators at the expected level.

Figure 2 - The local SLM reaches or approaches business targets on all KPIs, with an almost zero hallucination rate (0.4%).

On a representative file, the segmentation achieves a Cut F1 of 98 % : predicted boundaries almost all coincide with reference boundaries, with only two minor over-segmentations out of 48.

Figure 3 - Reference and predicted boundaries on a 79-page file. 46 of the 48 predicted boundaries are accurate; the two encircled over-cuts are the only discrepancies.

Speed: faster than the cloud

Direct measurement on the same 530-page file :

	Azure (DI + GPT-4o)	Local (GPU L40S)	Ratio
Wall-clock time	45 min	30 min	× 1.5 faster
Run cost	$8.70	~$0.45 (estimated based on hourly cost)	× ~20 cheaper

Table 6 - Azure vs. local on the same 530-page file.

The local cost is not zero: it is estimated by relating machine time to the hourly GPU cost (~$0.90/h). A 30 min run therefore costs ~$0.45 in compute, versus $8.70 charged by the cloud for the same file.

Figure 4 - DATES and SPLIT account for 92% of processing time: these are the two items to parallelise as a priority.

Note a limitation: a single GPU cannot handle an enormous parallel load: we cap out around ~15 pages/min at maximum load. This is perfect for asynchronous processing with moderate volume, less suited to a massive real-time peak.

Cost: the real argument at scale

The most significant gain is on cost. The GPU costs a fixed amount (~$0.90/h, or ~$650/month 24/7), independent of volume. The cloud, however, charges per page.

Metric	Azure	Local (GPU 24/7)
Marginal cost / file (530 p)	$8.70	~$0.45 (on-demand)
Fixed cost / month	0 (pay-as-you-go)	~$650
Break-even point	-	from ~75 large files / month

Table 7 - Cost - Azure vs. local GPU 24/7.

In 24/7 mode, the marginal cost of an additional file tends towards zero (the machine runs anyway). Beyond the threshold, any additional volume is processed at almost zero marginal cost on the AI infra side, and the gap widens linearly.

Figure 5 - Cloud cost grows with volume; on-premise GPU remains fixed. Beyond ~75 large files/month, the gap widens in favour of on-premise.

When to switch to an on-premise SLM

This migration is not a dogma. Here is the decision framework we derive from it.

Switch to a self-hosted SLM when:

you have a sovereignty / compliance constraint strong (sensitive data, regulated sector);
your volume exceeds the break-even point (typically a few tens to hundreds of files/month);
you regularly encounter API rate limits : self-hosting means regaining control over your throughput instead of being subject to the provider's quotas;
your use case is “classic” (OCR, segmentation, structured extraction, classification), where SLMs have caught up with the larger models;
your load is asynchronous and tolerates latency in the order of a minute.

What we've learned along the way

Quantisation is a local trade-off, not a recipe: the right level depends on the model, the task, and your VRAM.
MoE = quality of a large model at the throughput of a small one : an excellent profile for self-hosting.
Structure the task rather than letting the model reason freely: carefully crafted system prompt, and adversarial LLM-as-a-judge passes when quality is critical.
Test your guardrails in real-world conditions : open-source models hallucinate; the challenge is to detect and recover, not to aim for illusory zero defects.
Separate mechanical (reproducible) evaluation from semantic (LLM) judgment for KPIs auditable - and anonymise before any review by an external judge.

Conclusion

For this document processing, an open-source SLM in MoE architecture associated with a specialised OCR, on a single GPU at ~$650/month, achieves cloud quality, is faster, costs up to 20 times less at scale, and keeps sensitive data within the perimeter. Open-source doesn't replace everything: it's about choosing the right model for the right use case. For classic, high-volume tasks, under sovereignty or rate limit constraints, SLMs are ready.

By Maceo Duriez, AI Engineer at Galadrim. Feedback from a real client project.

FAQ

Can an open-source SLM match GPT-4o for document extraction?

Yes, for classic tasks (OCR, segmentation, structured extraction). On this project, the local MoE achieves the level of GPT-4o for segmentation and date extraction. Proprietary APIs maintain the advantage for advanced reasoning and complex agentic tasks.

At what volume does self-hosting become profitable?

Typically a few tens to hundreds of files/month. The GPU costs a fixed amount (~$650/month 24/7); beyond the break-even point (~3000 pages/month here), each additional file is processed at near-zero marginal cost, whereas the cloud charges per page.

How much does it cost to process a file locally vs. cloud?

For a single 530-page file: ~$0.45 in local compute (30 min at ~$0.90/hr GPU) versus $8.70 (approx. 16 cents/page) charged by the cloud.

When to stick with a proprietary API rather than an on-premise SLM?

When the volume is low, you need the most advanced reasoning, you need to handle massive load peaks without provisioning infrastructure, or the data has no localisation constraints.

Do you want support to launch your digital project?

Submit your project now