Can a small model replace the big LLM that labels evaluation reports?

Yes, with a big it depends. We scored 55 variants of 9 open models for EvalExplorer. A fine-tuned 2-billion-parameter model, about 60 times smaller than the pipeline's 117-billion-parameter LLM (gpt-oss-120b), gives the same labels on 85% of fields on average, and as a 4-bit file fits on a laptop. On the common labels it is reliable. On rare and loosely defined ones, no model we tried does well, and the reason is the training data, not the model.

{
  "evaluation_approach": "mixed_methods",
  "evaluation_type": "impact_evaluation",
  "temporality": "endline",
  "themes": [
    "global_health",
    "gender_equalities"
  ],
  "countries": ["MM", "UG"]
}

Frugal by design. Training the 2B model is one GPU for 21 minutes, under $1. Labelling then takes 0.4 to 1 second per report on one A100, a single GPU instead of a large hosted model. Small enough to run on your own machine: on a MacBook Pro (M5 Max), the 350M model as an 8-bit GGUF labelled the 134 test reports in 55 seconds (score 79.8). The 2B and 4B files have not been timed on a laptop yet, and energy use was not measured.

The question

When an evaluation report enters EvalExplorer, the ingestion pipeline sends its first pages to a large LLM (gpt-oss-120b, with Gemini 2.5 Flash and Qwen 3 235B as fallbacks). It returns five labels: the evaluation approach (mixed methods, experimental, ...), its type (impact evaluation, systematic review, ...), its timing (baseline, midterm, endline), its themes (global health, governance, ...) and the countries it covers.

How small can a model be and still give the same answers, so that this runs on a laptop or cheaply at scale, without calling a big LLM for every report?

The labels

Each report gets five fields. The codes and definitions below are the ones every model was given; the bars show how often the pipeline used each code across the 1,420 reports.

What we did

1. Took 1,420 reports the pipeline had already labelled: 1,148 to train on, 134 kept aside as the test.
2. Fine-tuned 9 small open models (350M to 26B parameters) to copy the pipeline's answers, with LoRA, and for two of them reinforcement learning (GRPO) on top.
3. Scored 55 variants in all (zero-shot baselines, fine-tunes, GRPO variants and GGUF exports) on the 134 test reports: how often does each give the same labels as the pipeline?

Results: best run per model

Score: agreement with the pipeline's labels on the 134 test reports, 0 to 100 (per report, 1 or 0 for approach, type and timing, F1 for themes and countries, then the average). Differences under about 3 points are within noise for 134 reports. "vs 3-LLM majority" is explained below. Every run, including the ones not shown, is in the experiments repo.

Run it locally

The strongest adapters exported to GGUF for llama.cpp. The 8-bit files match the original models; 4-bit costs a point or two for the 2B models and almost nothing for Qwen3.5-4B.

Gemma 4 26B-A4B scores lower as GGUF than its original run because its adapter behaves differently in plain transformers than in Unsloth, where it was trained and first scored; details in the GGUF repo.

Train your own

What one model costs on Hugging Face Jobs, measured from the jobs that produced the results above (A100 at $2.50 an hour, H200 at $5). Each job also scores the 134 test reports; the training data is 1,148 labelled reports.

For a similar task of your own: about a thousand labelled examples, the same recipe and scripts (code/jobs/sft.py, grpo.py, gguf.py in the experiments repo), and a few dollars per model. The whole study here, 55 variants of 9 models, cost about $45; the label-quality follow-on added about $26 of LLM relabelling.

Where it works and where it doesn't

The score above is an average over five fields, and it hides where the model fails. Only about 1 report in 4 has all five fields right. Below, every code: how many training examples it had, how far the pipeline and three newer LLMs agree on it (a measure of how well defined it is), and how often Qwen3.5 2B (SFT + GRPO, countries reward, lr 5e-6) finds it on the test set.

"Found": recall of the model on the test reports. "Labellers agree": F1 between the pipeline's labels and the 2-of-3 majority of GLM-5.3-Flash, DeepSeek-V4.1-Flash and Qwen3.8-2.4T-A95B over all 1,420 reports. Recall is measured on the 134 test reports; with fewer than about 10 test examples ("Test") it is a rough figure. "Train": training examples. Reliable: found at least 80% of the time and labellers agree at least 70%. Rare: under 60 training examples. Loosely defined: labellers agree under 60%.

Why: the labels, not the model

• Some codes are loosely defined. Each code has a one-line definition, and some overlap almost word for word: economic_development is "development finance, infrastructure", international_finance is "development finance, private sector". Even with 287 training examples, the pipeline and the newer LLMs agree on economic_development only 31% of the time. A model cannot learn a distinction its labels do not make consistently.
• The fuller definitions never reached the labels. Our original taxonomy has full definitions and long keyword lists per theme (social development alone covers social protection, cash transfers, children and youth, and social cohesion). The pipeline used one-line summaries of them. And the full taxonomy overlaps in places: growth and economic development share their trade and economy keywords, and nutrition sits under both food and agriculture and global health.
• Some codes are rare. developmental has 16 training examples, civil_society 27, rapid_evidence_assessment 34. Clear rare codes are learned (nature_environment, 32 examples, labellers agree 85%); rare and loosely defined ones are not.
• "Blank" is inconsistent. When to leave a field empty differs between labellers, so the models learned to almost never leave the type blank.
• The models only saw code names. The training prompt lists the allowed codes without definitions; the models learned what each code means from examples alone.
• The labels are LLM output. We treated the pipeline's labels as the gold set. Three newer LLMs agree with each other far more than with the pipeline, but on the 36 hand-checked reports, mostly evidence reviews, all three leave the approach blank where people gave one. Agreement is not correctness.

Mean field score between two label sets over all 1,420 reports. The pipeline is a 2025 model; the three relabellers are 2026 models given the same pages and code definitions. Details: the follow-on.

Data preparation is the work

What we would do before relying on the rare and loosely defined labels, in order:

1. Use the full taxonomy, and fix its overlaps. Bring the complete definitions and keyword lists into the labelling prompt and the model's prompt, resolve the codes whose keyword lists overlap, add an example and a counter-example for each neighbouring pair, and write a rule for when a field is blank.
2. Have people verify a sample. A few hundred reports, weighted towards the codes that are rare or loosely defined, so there is a gold set to measure against.
3. Collect enough examples of every code. Keep the rare codes; aim for at least 50 verified training examples each, and a test set with 20 to 30 per code so each one can be measured.
4. Then retrain. At $1 to $3 per model, this is the cheap step.

Next steps

• We are not putting this model into production yet. It needs more and better data first, so we will keep experimenting as more reports come in, until it scores high enough on every code, not just on average.
• Build the fuller codebook and a human-verified training and test set as more reports come in.
• Apply the same approach to excerpt tagging: findings, recommendations and methods inside each report, not only the whole document.
• Time the 2B and 4B files on a laptop, and measure energy use.

Read more

All runs

All 55 variants of 9 models, scored on the 134 test reports: zero-shot baselines, fine-tunes, GRPO variants and GGUF exports. Click a column to sort.

Score, vs GLM and vs majority: mean field score, 0 to 100, against the pipeline labels (the training target), the GLM-5.3-Flash relabelling and the 3-LLM majority. Approach, type and timing: accuracy; themes and countries: micro F1. Exact: all five fields right.