NVIDIA has released a small model with an idea I find genuinely fun to think about: Nemotron Labs Dory1.0-4B lets you decide, at inference time, how deep the network should think. One checkpoint, one dial, and a handful of settings. Here is what it is, what NVIDIA reports, and what I would want to measure before trusting it in an agent.
What looping means
A normal transformer is a tall stack of different layers. Your tokens go up through the stack once, and the answer comes out the top. If you want a smarter model, you usually make the stack taller or wider, which means more parameters and more memory.
A looped model takes a different path. It reuses the same block of layers several times. Think of it like reading a hard paragraph twice. You give the same brain another pass at the same material. The extra thinking happens in the model’s internal activations (what the model card calls latent space) rather than in extra words written out.
According to the model card, Dory splits its layers into three groups: 8 input layers, a shared recurrent block of 20 layers, and 8 output layers. The recurrent block is applied multiple times, and the input is reinjected into every loop. The number of loops, from 1 to 4, is chosen when you run the model. NVIDIA describes this as elastic inference compute: a single checkpoint serving different budgets, with no retraining and no model swap.
For developers, that is the interesting part. A quick classification call can use one loop. A gnarly coding task can ask for four. Same weights, same files on disk.
Memory stays flat, in one mode
The usual worry with looping is the KV cache, the memory a model keeps for the tokens it has already seen. If each loop stores its own cache, memory grows with every loop you add.
NVIDIA says Dory supports a last-loop KV cache mode. Each loop overwrites the previous loop’s entries, so only the final loop’s keys and values are kept. NVIDIA states that this carries no KV cache overhead compared with an equivalent non-looped model, whatever the loop count, and that accuracy stays close to the per-loop mode in its experiments. The default examples in the card use per-loop caching, so you opt in to the smaller footprint by setting recurrent_kv_cache_mode to "last_loop".
The other headline specs from the card: 4B parameters, up to 256K tokens of context, English only, and the OpenMDW License version 1.1. NVIDIA says it pretrained on 6T tokens, the first 4.5T or so natively in its NVFP4 four-bit format.
NVIDIA’s numbers
All of these come from NVIDIA’s own evaluation, run under its harness, so treat them as the vendor’s claims. On SWE-bench Verified, the card lists:
- Dory1.0-4B at 1 loop: 50.3
- at 2 loops: 59.5
- at 4 loops: 61.8
- Qwen3.5-4B: 54.7
- Qwen3.5-9B: 61
So in NVIDIA’s table, two loops pass the 4B Qwen model, and four loops edge past the 9B one. NVIDIA recommends two loops as the starting point and says most of the gain arrives with the second loop. The table is not a clean sweep. On GPQA Diamond, for example, Dory at two loops scores 72.41 against 77.71 for Qwen3.5-4B. NVIDIA also says higher loop counts can reduce the number of generated tokens, which matters when you think about end to end latency.
Running it
Support is still arriving. The card says Dory support in vLLM is under review in vllm#60149, with instructions to install from that pull request until a release includes it. For llama.cpp, the card points to ggml-org/llama.cpp#30306 and gives a Dory-enabled build recipe, along with a Q5_K_M GGUF export. NVIDIA also lists BF16, MXFP8 and NVFP4 checkpoints, and reports DGX Spark speedups over BF16 of 2.17x to 2.44x for NVFP4 in its two test configurations.
One related note for vLLM users. The v0.32.0 release says NVFP4 KV cache now runs on SM8x and SM12x, which covers consumer RTX cards, and it adds a generic hf tool-call and reasoning parser that reads a checkpoint’s response template, so models that ship one need no model-specific parser. The release page does not mention Dory, and the Dory card still uses its own parser flags (qwen3_coder for tools, nemotron_v3 for reasoning), so follow the card for this model for now.
What I would check first
A few honest caveats before anyone rewires an agent around this:
- The benchmark figures are NVIDIA’s, on NVIDIA’s harness. Independent runs would help.
- Looping multiplies the compute for each token. NVIDIA suggests fewer generated tokens may offset that, and I think that is plausible, but I have not seen laptop latency at 2 to 4 loops measured anywhere, so I would not promise a number.
- Deployment lags launch. Needing a pull request build of vLLM or llama.cpp is fine for experiments and awkward for production.
- It is English only, and the card lists NVIDIA Hopper and Blackwell GPUs on Linux as its supported hardware, so your mileage on other setups may vary.
Where I would start
If I were trying it, I would serve it at two loops, run my own small task set at 1, 2 and 4 loops, and watch three things: quality, total latency per finished task, and memory with last-loop caching on and off. Your own workload will tell you more than any table.
Models like this change what “small” can mean, because depth becomes something you budget per request. Before you build on it, open the model card and the runtime docs and read them end to end. The details on caching modes, loop settings and supported hardware are all there, and they are worth your time.