Research and benchmarks¶

This page collects Wren's architecture, measured results and research studies. For installation and use, see the How-to.

Architecture¶

Wren is an attention-only byte-level decoder: transformer blocks without the feed-forward layers, with the parameter budget spent on attention instead. The prompt supplies target names and current values; the model generates set and read calls. Attention projections use ternary weights, packed for storage; embeddings and normalization scales remain float32. NumPy and C runtimes load the same pack.

Demo installation¶

The shipped model was trained on an earlier demo home with 20 targets, 50 examples and 18 tests, using fit's shape and step defaults: 256 wide, 4 layers, 4000 steps with batches of 16. The current wren/home.json contains 56 examples, 8 calibration records and 26 tests; its retrained candidate was not promoted, as described below.

These historical generated evaluation samples used another seed and the same templates and examples as training; command wording could overlap. The current evaluator separates calibration and holds out phrasing families. Exact match compares target, value and delay, ignoring call order and confidence. Coverage counts samples handled exactly right after applying only calls at or above the threshold, including correct abstentions. Two of the five covered tests are abstentions. These measurements describe this pack, not expected accuracy for a new installation.

A fresh CPU audit measured 4/18 covered tests: "stop the sprinkler" scored about 1.2e-7 below the saved threshold. The historical 5/18 result is sensitive to this small numerical difference.

The threshold sits just above the most confident wrong answer to a test: "kill all the lights" switches them all on, at 0.96. On generated commands alone the threshold would be 0.74.

The demo knows only its 20 targets, by their names and descriptions, and English commands in the generator's style. Words the generator does not use are where it fails: "kill" is not a word it knows for off, "half an hour" is not a delay it knows (it opens the awning now, at 0.68), and "good night" switches the bedroom light off (at 0.51). It also miscopies the number in about a third of the delays ("in 87 minutes" as 88), always below the threshold: at the threshold it handles 2 of the 97 delayed commands. Of a duration ("off for 7 minutes") it often applies the call for now but not the one that ends it.

The s12 study compares the shapes and step budgets behind the training defaults.

Accuracy improvements, 2026-10-25¶

s13 adds separate calibration (drawn like the training data, so most of its commands were seen in training), complete-command metrics, negation/timing data, whole-answer acceptance and selection audits. New packs default to accepting every generated call together or rejecting the answer. The shipped pack's policy is unchanged.

A CPU retrain used the existing 256×4, 4000-step stream recipe with the expanded data. Both packs were evaluated on the same 1000 unfiltered held-out samples and 26 installation tests, with whole-answer thresholds chosen on separate calibration data.

The candidate correctly rejected both tested negations and returned the right delays for all six added timing tests. However, it accepted "kill all the lights" as on, and accepted no correct positive installation test. It therefore did not replace the shipped pack. Calibration precision alone did not establish reliability on independent commands.

The selection audit retained 18/1000 generated failures: nine missing required targets and nine answers exceeding the byte limit. Installation tests had full target and declared-state recall.

The training-objective pilot compared stream windows, intact examples and intact examples with prompt loss weight 0.1 across three initialization seeds. At 64×2 and 500 steps, mean held-out exact match was 12.5%, 21.8% and 25.8%, respectively. These early gains justify a full-size comparison; they do not justify changing the production training recipe. The study README records settings, run IDs, limitations and reproduction commands.

Historical mobile-actions benchmark¶

Comparison with needle 2 on google/mobile-actions: all 961 eval rows, ordered strict exact match as in needle's protocol, greedy decoding. Wren trains on the 8,693-row train split plus about 2% synthetic negatives for 8000 steps in s7 and s8. These studies use historical models and APIs rather than the current demo.

¹ JAX CPU runtime on x86, prompt processing included; most of the RAM is the JAX runtime. ² runtime/wren.c, single-threaded on x86. ³ Raspberry Pi 5, from the needle 2 page. These throughput measurements use different hardware and units.

Needle has tool schemas in context, grammar-constrained decoding and a 115B-token pretraining corpus. Wren learns the seven tool functions in its weights and decodes unconstrained after training only on the task data.

Study index¶

Each study in dev/ documents its question, findings and reproduction commands.

Reproduce studies¶

Study s12 runs with the current target API; its published measurements used the older demo data at tag v1.1.0. Its constant learning rate also differs from fit's current cosine decay.

Studies s1–s10 document historical models and data. Their READMEs target v1.1.0, which retains the home API removed in Wren 2.0. Use a separate checkout for those commands:

git clone --branch v1.1.0 https://github.com/stroblme/wren.git wren-v1
cd wren-v1
uv sync --locked --group cuda12
uv run --group cuda12 fluksio serve --gpus 1

Training those studies requires a compatible NVIDIA GPU and driver. In a second terminal at the study's checkout, follow its README. Use Fluksio to track experiments; data downloads go into ignored data/, and plots into ignored dev/*/figures/.

Data¶

Studies download their benchmarks into data/: google/mobile-actions (CC-BY-4.0) and acon96/Home-Assistant-Requests (MIT). An installation's model trains on commands generated from its own target list (wren/dataset.py).