Problem
Cloud LLM benches tell you almost nothing about a 4 GB Pi, an 8 GB Jetson, a 16 GB Mac mini, or a phone on battery. Vendor slides quote peak tok/s under conditions you will never run. Cooling, power modes, and backend choice move the numbers more than the model card does - and most write-ups skip that.
Vision
A readable resource you can actually consult: which model fits which board, ranked by metrics that matter on-device: tok/s when you care about speed, tok/J when you care about power.
The leaderboard is the simple table. Same prompts, same load generator, power and thermals recorded, caveats written down. Focus is everyday consumer hardware that rarely shows up in LLM benches but has real potential: Mac Minis, Jetsons, Raspberry Pis, tablets, and phones.
Need the full story? Each device has detailed benchmark reports. Start with Jetson here; Pi, Android, tablets, and Mac Mini land as those runs publish.
What we do
smolperfbenchmark is the public leaderboard for the smolperfbenchmark harness: self-contained per-device folders - scripts, chart generators, and reports - not a single mega-framework. Each device folder owns its harness (aiperf). This page is the public cut of those results
- Tiny instruct models that actually load on consumer boards (≈135M-8B class).
- Headline metric:
output tok/J- watts matter when the board is the budget. - Backends compared fair: same GGUFs from llama.cpp through Ollama and MLX-LM etc where they run.
- Raw artifacts published to Hugging Face with detailed write-ups on blogs.
For the harness, sweeps, and how to reproduce a run, see How it’s done.
