How to build a small LLM: Sheared-LLaMA's 3% compute path

A small model carved out of a big one beat same-size models trained from scratch, at under 3% of their compute. Sheared-LLaMA prunes the open LLaMA2-7B down to 1.3B and 2.7B, continues pretraining on about 50B tokens, and surpasses Pythia, OPT, TinyLlama and OpenLLaMA, models trained on 300B to 3T tokens, on average downstream performance ^[1]. If a company without its own LLM wants to build a small LLM, the cheapest route runs through this paper.

This article reads the ICLR 2024 camera-ready of arXiv 2310.06694: the abstract, section 2, Tables 2 and 5, and Algorithm 1, as of November 2026; code and models are in the public repository and on Hugging Face ^[1]. The claims about agent workloads come from a position paper by NVIDIA researchers; it is not an experimental paper, and its replacement ratios are author estimates ^[2]. We reproduced neither, and we have no relationship with the authors.

Why not train a small LLM from scratch

For the same model size, the starting point changes the cost by orders of magnitude. Public small models trained from scratch carried token budgets of 300B for Pythia, 800B for INCITE, 1T for OpenLLaMA and up to 3T for TinyLlama ^[1]. An open model that has already consumed trillions of tokens holds the knowledge that training compressed into it, so carving it down starts small while inheriting that knowledge. The obstacle was that existing pruning degraded performance heavily, with no settled recipe for recovering it ^[1].

Sheared-LLaMA's answer has two stages: targeted structured pruning, which fixes the output architecture in advance and cuts toward exactly that shape, and dynamic batch loading, which recovers the pruned model efficiently on a small token budget. Pruning is only an initialization; most compute goes to recovery through continued pretraining ^[1].

Figure 1. The two stages of Sheared-LLaMA: targeted structured pruning (0.4B tokens) cuts the 7B model to a target architecture, and continued pretraining on about 50B tokens recovers it while dynamic batch loading adjusts domain proportions ^[1].

What is LLM pruning: targeted structured pruning

LLM pruning shrinks a model by removing structural units: layers, attention heads, hidden dimensions. The variant in Sheared-LLaMA differs in that the model chooses where to cut, while constraints force the result to be exactly the target architecture. A mask $z$ marks each structural unit as kept or removed, and the objective minimizes the language-modeling loss of the masked model.

Here $\theta$ is the model weights, $z$ the pruning masks over layers, heads, intermediate and hidden dimensions, and $\varphi$, $\lambda$ are Lagrangian terms that force the masks to converge to the target shape (Pythia-1.4B's architecture for the 1.3B model, INCITE-Base-3B's for the 2.7B) ^[1]. Controlling the output shape, rather than cutting into an arbitrary one, means the result inherits an architecture whose inference efficiency is already proven. This stage uses only 0.4B tokens; the authors report that within a fixed budget, more pruning tokens improve the recovered loss, but pruning is computationally expensive per token, so 0.4B is the practical trade-off ^[1].

Dynamic batch loading: the loss decides what data to feed

A pruned model recovers at different speeds across domains. The authors fit a scaling law to the LLaMA2 series to predict the per-domain loss a model of this size would reach if trained from scratch, take that as the reference ${\ell }_{\mathrm{ref}}$, and upweight the domains furthest from it ^[1]. Every $m$ steps the gap for domain $i$ is measured and the proportions update.

${\ell }_t[i]$ is the current validation loss on domain $i$, and $w_t$ is the proportion of each domain in upcoming batches, renormalized to sum to one; domains with larger gaps are sampled exponentially more ^[1]. Feeding RedPajama at fixed proportions recovers domains unevenly and wastes data, and this adjustment lifted downstream performance within the same token budget ^[1].

Results: 50B tokens beat 300B to 3T

The two models, built with about 50B tokens in total (0.4B pruning plus continued pretraining), surpassed same-size public models on average downstream performance ^[1]. Copying only the averages from the paper's Table 2:

Table 1. Downstream averages against same-size public models, with training tokens in parentheses. Sheared-LLaMA posts the highest average in both size classes with the fewest tokens ^[1].

In cost terms, the authors report reaching parity with OpenLLaMA-3B-v2 on 1/32 of its budget, about 3% ^[1]. On instruction-tuning evaluations the models also won against same-size public models ^[1]. NVIDIA later carried the recipe into practice with Minitron, combining pruning with knowledge distillation to shrink Llama 3.1 8B to 4B ^[3].

Are small models enough for AI agents

Being able to build one is separate from it being enough. The position paper from NVIDIA researchers argues that agentic systems decompose complex goals into modular subtasks, and that most of those subtasks are narrow, repetitive and non-conversational, which makes small models sufficient and more economical ^[2]. On numbers, it cites serving a 7B model as 10 to 30 times cheaper than a 70B to 175B model in latency, energy and FLOPs, and, dissecting three open agents, estimates the share of LLM calls replaceable by small models at about 60% for MetaGPT, 40% for Open Operator and 70% for Cradle ^[2]. These are author estimates in a position paper, but the direction matches what we saw in the analysis of AI search agent queries: most of what agents actually issue is narrow work, classification, extraction and format conversion.

Stack the two papers and the conclusion follows. If most agent workload sits at small-model level (the capability claim ^[2]), and owning that small model now costs around 3% of from-scratch training (the cost claim ^[1]), then a company without its own LLM has a real option to own the models behind its orchestration layer.

Limits

What the papers and this article do not cover comes first.

• The artifacts are 2023-era. The 1.3B and 2.7B models themselves trail later small-model generations trained on tens of trillions of tokens. What survives is the method, not the checkpoints; applying it today means pruning a current open-weight model, and the practical recipe is Minitron ^[3].
• 50B tokens is not free. Three percent of from-scratch still means multi-billion-token training infrastructure. Against SmolLM2's 11T tokens from scratch it is small, but it is not a laptop job ^[4].
• The license carries over. Pruning LLaMA2 keeps you inside LLaMA2's license terms. Check the base model license before commercial deployment.
• The reference loss is a scaling-law estimate. Dynamic batch loading relies on a function fitted to the LLaMA2 series to predict a hypothetical small model's loss; if that estimate is off, the reweighting is off ^[1].
• No direct agent evaluation. Table 2 covers commonsense, reading and knowledge benchmarks; tool calling and structured output sit outside the paper's scope. The position paper's estimates fill that gap for now, so adoption should be measured on your own workload.

In practice: a cost baseline for building an sLLM

What this paper gives practice is a baseline for the build-vs-buy call. Classify every frontier-model call in an agent pipeline and, outside complex generation and coding, most calls are repetitive subtasks: classification, extraction, validation, format conversion ^[2]. That segment favors an owned small model over an API bill the more ① call volume grows, ② latency matters and ③ the output format narrows. Segments with low volume or long-tail reasoning still favor the API. The pattern we see operating content-generation pipelines is the same: the narrow stages around generation account for most of the calls.

We have seen this asymmetry elsewhere. The world model that teaches a robot from video learns the world from the data-rich side (internet video) and attaches the data-poor side (robot actions) thinly; the pruning route shares that structure, starting from the knowledge-rich side (the trillions of tokens compressed into a 7B model) and bridging to the small architecture with few tokens. The principle that working in a compressed feature space wins also connects to structural feature engineering.

Read next

The practical pruning-plus-distillation procedure is laid out in the Minitron paper ^[3], and the from-scratch cost baseline is public in SmolLM2 ^[4]. For what agents actually ask, see the analysis of AI search agent queries; for video understanding in retrieval, see the video RAG paper review.