SimpleLMBenchmarks
A million tokens. Measured, not asserted.
Three long-context benchmarks, run at every length.
Fusion Aperture achieves state-of-the-art quality-memory efficiency among training-free long-context inference systems at the 2B Q4, mobile, half-gigabyte-class operating point.
Benchmark 01
RULER
Thirteen synthetic tasks covering retrieval, tracing, aggregation and question answering, averaged at every context length.
at 128K, and 72.0 at one million
Score and memory, one axis
Context grows 256×, from 4K to 1M. Both panels share the axis at the bottom, so read them vertically.
Score panel spans 50 to 100; memory is drawn from zero. Vertical ticks show the minimum to maximum spread around each median. Dashed segments span 128K to 1M, where nothing between was measured. The 767 MB maximum there is a brief spike, not a resting level.
Where the numbers land.
Left is the 128K column nearly every system reports. Right is one million, where the field thins out. The names that vanish between the two lists are the point.
At 128K
The length nearly every long-context system publishes.
- 01Jamba-1.5-large94B/398B95.1
- 02Gemini-1.5-pro94.4
- 03Qwen2.5-14B-Instruct-1M14B92.2
- 04Qwen3-235B-A22B235B90.6
- 05Qwen2.5-7B-Instruct-1M7B84.4
- 06GLM4-9B-chat-1M9B83.1
- 07GPT-4-1106-preview81.2
- 08Fusion Aperture2B80.3
- 09Qwen3-30B-A3B30B79.2
- 10Qwen3-8B8B77.4
- 11Llama-3.1-8B-Instruct8B77.0
- 12Llama-3-8B-Gradient-1048k8B69.5
- 13Phi-3-mini3.8B66.7
- 14Qwen3-4B4B66.0
- 15LWM-Text-Chat-1M7B65.0
Parameter counts shown beside each name. Other rows as published by their authors; sample counts and hosts differ between systems.
At 1M
Every system with a published RULER figure at one million tokens.
- 01MiniCPM-SALAtrained86.3
- 02Qwen3-235B-A22B-Instruct-2507trained84.5
- 03Qwen3-Next-80B-A3B-Instructtrained80.3
- 04Qwen3-30B-A3B-Instruct-2507trained72.8
- 05Fusion Aperturetraining-free72.0
Ours at ten samples per task; the Qwen rows at twenty. Four other systems have published a 1M figure. That is the whole list, and every one of them was trained for it.
Most stop at 128K.
Systems working on the same problem, each on the host it published against. The empty column on the right is not a criticism. It is simply what exists.
- MiniCPM-SALAtrained hybrid 9B89.486.3
- MagicPIGLlama-3.1-8B-Instruct81.7@96KNot reported
- Fusion Aperturestock Qwen3.5-2B Q480.372.0
- MInferenceLlama-3-8B-262K77.6Not reported
- QuestLlama-3.1-8B-Instruct74.9@96KNot reported
- Full attentionLlama-3-8B-262K72.2Not reported
- InfLLMLlama-3-8B-262K39.5Not reported
- StreamingLLMLlama-3-8B-262K9.4Not reported
Every row except MiniCPM-SALA is a training-free inference method. What differs is the host: every other row was measured on a long-context-extended 8B to 9B model at full precision. Two rows report at 96K rather than 128K, marked inline.
Benchmark 02
Needle in a haystack
One fact buried in a long document, at eleven depths across ten context lengths. Every cell up to 512K was found.
107 of 110 cells
110 cells, three holes
One fact hidden at eleven depths, across ten context lengths. Every cell up to 512K was found.
Depth of insertion runs top to bottom, context length runs left to right
- Overall
- 97.27%
- 107 of 110 cells
- 1K to 512K
- 100%
- 99 / 99 cells
- At 1M
- 72.73%
- 8 / 11 cells
- Missed depths
- 3
- 33.91%, 66.09%, 89.57%
Memory during the run
461 MB
Peak, at 1M
One sample per length. A reading, not a sweep.
There is no leaderboard here.
Unlike RULER, this benchmark has no shared board to join. Most systems publish a coloured heatmap and no figure at all. Ours sits next to what everyone else actually released.
- Fusion Aperture72.73% at 1M, 97.27% overall
- Gemini 1.5 Pro>99.7% single-needle at 1M, about 60% on 100-needle
- MInferenceHeatmap only, no overall figure
- StreamingLLMHeatmap only, middle collapses
- InfLLMHeatmap only, middle collapses
- MiniCPM-SALANo 1M figure published
These are not the same protocol. Ours is an eleven-depth mean over essay text; the Gemini figures come from a different needle procedure. Placed side by side for context, not as a ranking.
Benchmark 03
MRCR v1
Multi-round conversations where the right answer depends on which earlier turn you meant. Ninety-one of them, in the longest bin the benchmark publishes.
2-needle, 512K to 1M
2-needle at 1M
Every published figure on this board. Seven thousandths separate us from GPT-4.1 mini.
- 01MiniMax-M1-40k0.586
- 02MiniMax-M1-80k0.562
- 03GPT-4.10.463
- 04GPT-4.1 mini0.333
- 05Fusion Aperture0.326
- 06GPT-4.1 nano0.120
Scores are similarity ratios between 0 and 1. Other rows as published by their authors.
Memory during the run
520.6 MB
Median
Across 69 samples. The peak sits above our 500 MB line, so we print it rather than round it.
How to read these numbers.
Everything on this page is either a run we can point to, or a figure someone else published with a source.
Our figures
RULER is ten samples per task at each length. NIAH is a single pass over all one hundred and ten cells. MRCR is ninety-one conversations in the 512K to 1M bin.
Everyone else’s figures
Cited, never re-run by us. Sample counts, hosts, and harnesses differ between systems, which is why we compare scores and never memory. Sources are listed below.
Memory
Reported as minimum, median and maximum across the run. The median is the figure worth reading; peaks are brief. We compare our own memory across lengths, not against other systems.
Empty cells
Where a system has not published a figure at a given length, the cell says so. We do not interpolate, extend a line past its last measurement, or substitute a nearby length without marking it.
The host
The host is a stock Qwen3.5-2B Q4_K_M, with unmodified weights and no long-context fine-tuning.