neural voxel parkour

by Ryan Alport Jun 19, 2026

the idea

I wanted a small 3D voxel parkour game where the physics step could be swapped at runtime. The analytic engine handles acceleration, friction, gravity, jumping, and axis-separated AABB collision in one function: physics_step(). The goal was to train a neural network that predicts the same state transition, then push inference latency down until it is close to the analytic baseline.

That means two parallel optimization tracks. On the algorithmic side: keep the teacher fast and deterministic. On the neural side: shrink the observation, quantize weights, fuse normalization, and tune the matmul kernels until the gap is measured in single-digit microseconds, not milliseconds.

Target accuracy is at least 98% one-step grounded agreement on held-out data. Every model we kept clears that bar. The harder metric is rollout: run the network for hundreds of ticks and compare to analytic physics. That is what you feel in-game as drift, jitter, and phasing through voxels.

The current default is model_q8_h24_p3.bin: 39 inputs (3x3x3 voxels + 12 scalars), 24-neuron int8 MLP, ~0.95 us per neural step (~12x analytic at ~0.08 us). It is not the fastest option we trained, but it is the most stable on long runs among the models we still ship: ~89% rollout grounded agreement, ~2.4% of airborne steps overlap solids (analytic: 0%).

Most of the code — C inference, training pipeline, benchmarks, int8 export — was built in one afternoon with Cursor agents from a written spec. See implementation and building with cursor below for file layout, the record→train→bench loop, and what made iteration fast.

the game

16x16x16 voxel world, procedural parkour maps, orange cube player, WASD and space to jump. Each launch generates a new map via random-walk platforms with gaps and height changes. Blue start, green goal. Fall below y = -4 and respawn. Reach the goal and the world regenerates.

screenshot from the raylib build. gray platforms, blue start, green goal, orange player. Tab toggles between analytic and neural physics mid-game.

architecture

The physics pass is isolated behind a single call site so analytic and neural paths share the same game loop:

if (physics_mode == PHYSICS_ANALYTIC)
    physics_step(&player, input, dt);
else
    neural_physics_step(&player, input, dt);

The analytic engine is the teacher. During data collection it records paired before/after states under randomized input policies. The neural path loads a binary MLP from disk and runs inference in C with no pytorch at runtime.

Output is 7 floats: dx, dy, dz, next vx, next vy, next vz, next grounded. Training uses pytorch; inference uses exported float or int8 weights with per-row scale factors.

implementation

The repo is plain C99 + raylib, CMake, and a small Python training script. No engine, no ECS, no runtime ML dependencies. The whole project is small enough to hold in your head: seven C files, one header of shared constants, two Python tools.

code layout

closed loop

Everything hangs off one pipeline. Same binary, different CLI flags:

# 1. record 500k one-step transitions from analytic physics (headless, ~seconds)
./voxel_parkour.exe --record 500000 --out data/train.bin

# 2. train + export int8 models (~12 min stable sweep on CPU)
python tools/train.py --sweep-stable --epochs 40

# 3. measure rollout + latency vs analytic
./voxel_parkour.exe --bench models/model_q8_h24_p3.bin

# 4. play, toggle mid-game
./voxel_parkour.exe

Recording never opens a window. It runs the same physics_step() as the game, writes fixed-size rows to disk, and rotates maps when the player falls or an episode hits 400 steps. Training reads those rows, holds out 20% of map seeds for validation, and exports weights the C loader understands.

observation format

Each training row is 90 bytes, packed to match C struct layout exactly:

• 27 bytes — voxel types in the 3³ neighborhood (0=empty, 1=solid, …)
• 12 bytes — sub-voxel offset (3 floats)
• 12 bytes — velocity before the step
• 6 bytes — grounded flag + 5 input buttons
• 4 bytes — fixed dt (1/60)
• 25 bytes — analytic targets: Δpos, next vel, next grounded
• 4 bytes — map seed (for train/val split)

At inference, pack_player_features() builds the same 39-float input vector the trainer sees: voxels / 4, offsets, normalized vel, grounded, one-hot inputs. Train/serve skew here would show up immediately as rollout drift, so the packing path is shared logic duplicated once in C (not two different codepaths).

model binary format

Shipped models are int8 v2 blobs (~1–4 KB). Header: magic MLP!, version, four ints (input_dim, h1, h2, output_dim). Weights are per-row quantized int8 with float scales; biases are pre-fused with input/output normalization at export time so inference never touches mean/std arrays. Forward pass: int8 matmul → scale → ReLU → int8 matmul → scale → 7 outputs applied directly to player state.

void neural_physics_step(Player *p, InputState input, float dt) {
    float in[INPUT_DIM], out[OUTPUT_DIM];
    pack_player_features(p, input, in);
    neural_forward(in, out);
    p->pos.x += out[0];  p->pos.y += out[1];  p->pos.z += out[2];
    p->vel.x = out[3];   p->vel.y = out[4];   p->vel.z = out[5];
    p->grounded = out[6] >= 0.5f ? 1 : 0;
}

AVX2 FMA kernels live only in neural.c. Applying -march=native to the whole executable caused stack misalignment crashes in raylib's render path; scoping SIMD to the inference translation unit fixed that.

benchmark modes

--bench runs 50 episodes × up to 300 steps. Each step: same random input drives analytic and neural players in parallel from identical starting state. We track mean position/velocity error, grounded mismatch %, and tunnel count (neural AABB overlapping solids while airborne). That last metric matches what you see as phasing in play.

analytic teacher (what the network imitates)

The teacher is deliberately boring: axis-separated AABB collision against a 16×16×16 voxel grid, no rotation, no swept tests. Each frame applies input acceleration, horizontal friction, speed clamp, optional jump, gravity, then moves X → Y → Z independently — if a move collides, snap back and zero velocity on that axis. Landing on Y sets grounded = 1.

void physics_step(Player *p, InputState input, float dt) {
    /* accel + friction + jump + gravity */
    p->grounded = 0;
    move_axis(p, 0, p->vel.x * dt);  /* X, then Y, then Z */
    move_axis(p, 1, p->vel.y * dt);
    move_axis(p, 2, p->vel.z * dt);
}

Recording snapshots the player before the step, runs physics_step(), snapshots after, and writes the delta as training targets. The network never sees collision resolution code — it learns the end-to-end mapping from local geometry + state to the teacher's outcome.

training and export

train.py loads the binary dataset with numpy struct unpacking (one frombuffer pass over 500k rows, not a Python loop per sample). Map seeds split train/val 80/20 so the network generalizes across procedural layouts, not just held-out frames from the same map.

The MLP is a plain two-layer ReLU net in PyTorch: 39 → h → 7. After training, fuse_weights() bakes input/output z-score normalization into the first and last layer so C inference never loads mean/std arrays:

w1 = w1 / x_std;  b1 = b1 - w1 @ x_mean   # input norm → layer 1
w3 = w3 * y_std;  b3 = b3 * y_std + y_mean   # output denorm → layer 3

int8 export quantizes each weight row independently (per-row scale, max abs → 127). Biases stay float32. The C loader reads magic MLP!, dims, packed int8 weights + scales, and runs the same graph as PyTorch at eval time — within quantization error. Sweep flags train multiple hidden widths in one invocation, reusing the cached dataset between models.

build and entry points

CMake fetches raylib 5.5, builds one executable with Release defaults (-O3 -ffast-math). AVX2/FMA flags apply only to neural.c — not the whole binary. Same exe, four modes:

Model load tries several relative paths (models/…, ../models/…) so the same binary works from repo root or build/. Default search order prefers h24, then h48×24, h64, h48.

building with cursor

I did not hand-write most of this. I wrote spec.md — constraints, data formats, accuracy floor, swap interface — and drove the implementation through Cursor agents in a single long session over one day. My job was picking tradeoffs and reading benchmark output; the agent's job was generating code, running builds, fixing compile errors, and iterating until the numbers moved.

why this project fits agents well

• Closed loop with objective metrics. Record, train, bench, play. Every hypothesis ends in a number: microseconds, grounded %, tunnel count. Agents can run the loop without me context-switching into five tools.
• Small surface area. ~2k lines of C, one MLP, one training script. Fits in context. Refactors don't require understanding a whole engine.
• Spec as source of truth. Fixed dt, no pytorch at runtime, 98% one-step floor, binary export format — agents had concrete acceptance criteria instead of "make it feel good."
• Embarrassingly parallel experiments. "Try 7³ patch." "Add int8 export." "Sweep h12–h32." Each is a prompt + a bench run, not a week of manual porting.

spec.md as the contract

Before any code landed, I wrote a spec: observation layout, record format, model binary layout, accuracy floor, swap interface. That file became the agent's acceptance test. When rollout looked bad, I didn't argue about feel — I pointed at spec constraints ("no analytic fallback at runtime") and asked for new metrics. When the agent added collision fallback to hide drift, I rejected it against the spec and the real problem surfaced.

Concrete fields in the spec — 90-byte rows, 7-float outputs, fixed 1/60 dt — meant the agent could generate matching C structs, Python loaders, and export code in one pass without me hand-aligning offsets.

prompts that actually moved the needle

Short, outcome-oriented prompts worked best. The agent had shell access, so "make it faster" became "Release build, fuse norm, re-bench" in one turn. Examples, paraphrased from the session:

• "Implement the full pipeline from spec.md: record mode, train.py, C inference, Tab toggle." — greenfield in ~2 hours of agent time.
• "305 µs is too slow. Optimize inference without changing accuracy." — Release flags, fused norm, single-pass pack; ~50× in one iteration.
• "Try shrinking the patch to 7³, then 3³. Sweep and bench each." — touched common.h, crop logic, re-export, full metric table.
• "Remove analytic fallback — pure neural only." — one function edit, rollout truth exposed.
• "Training sweep takes forever. Vectorize the loader." — 500k-row parse dropped from minutes to 0.3 s.
• "Game segfaults on launch after AVX." — gdb trace, scoped SIMD to neural.c, no raylib rabbit hole.
• "Bench numbers don't match playtesting." — found sample_in[729] overflow in timing loop.

I rarely pasted stack traces or file paths. The agent searched the repo, ran commands, and reported numbers back. My input was mostly constraints and yes/no on tradeoffs.

what a typical iteration looked like

Prompt → agent edits C/Python/CMake → agent runs cmake --build → agent runs --bench or train.py --sweep → I read the table → next prompt. Examples from this project:

what I still had to do

Agents are fast at implementation; they don't replace judgment calls:

• Remove analytic fallback — agent added collision fallback when rollout looked bad. I said no: pure neural or nothing. That made the real rollout problem visible.
• Pick the default model — h12 is fastest; h24 is most stable on long runs. Agent would have kept optimizing latency; I chose rollout over raw µs.
• Native 3³ re-record — cropping center 3³ from old 9³ data was a hack. I asked for honest recording at PATCH_D=3. Accuracy similar, rollout different, old weights discarded.
• Read the game — benches don't capture feel. I playtested, reported phasing, agent updated the blog and pointed at tunnel metrics. Subjective + objective together.

iteration speed (rough)

where time actually goes

Cursor compresses typing and glue, not GPU/CPU training. A typical experiment loop:

• ~30 sec — agent edits 2–5 files, rebuilds Release binary
• <5 sec — re-record 500k rows if observation shape changed
• ~13 min — train + export 4 int8 variants (the real bottleneck)
• ~10 sec — bench all models, paste comparison table
• ~2 min — me: read table, pick next hypothesis, send one prompt

Nine patch/architecture pivots in one afternoon is feasible because steps 1, 2, 4, and 5 are agent-owned. I was never waiting on me to write matmul kernels or debug CMake. I was waiting on epoch 37 of four parallel MLPs — but that runs unattended while the agent logs ETA.

Compare to solo dev without an agent: same afternoon might cover "get pytorch export matching C struct layout" and one broken bench run. Here that was the first hour, and the rest was optimization and experimentation.

human vs agent split

End-to-end from empty repo to playable neural swap with documented benchmarks: one afternoon. Without Cursor that is easily a week of setup, training pipeline boilerplate, and inference export glue — work I didn't want to write by hand.

starting assumptions

These were the constraints I set before any optimization work began. They shaped every later decision.

• Fixed dt only. 1/60 s everywhere. No variable timestep in training or inference.
• Pure C MLP inference. No pytorch, no BLAS dependency at runtime. Weights load from a binary blob.
• Analytic teacher. physics_step() generates labels. The network learns to imitate it, not invent its own physics.
• One-step supervision. Train on single-tick transitions. Rollout drift is measured separately; it now drives default model selection, not just diagnostics.
• Swappable call site. Same game loop, same inputs, same observation packing. Only the step function changes.
• No collision correction at runtime. The network sets position and velocity directly. There is no analytic fallback. Small per-frame errors compound; the player can end up inside voxels.
• 98% grounded floor (one-step). Any model below that on held-out validation is rejected. Rollout stability is a separate, harder filter for picking the default.

Full writeup of the original spec lives in spec.md in the repo. See building with cursor above for how those constraints were enforced in practice.

current models

Four int8 models remain in models/. All use native 3x3 observations (500k recorded samples, 90-byte rows). Older 7x7, 9x9, float, and speed-ladder (h12/h16/h32) checkpoints were removed after rollout testing.

*tunnel = fraction of steps where the neural player overlaps solid voxels while not grounded. Analytic physics: 0%. Measured on 50 episodes x up to 300 steps with --bench.

optimization progression

Each checkpoint below is a real model variant we trained and benchmarked. For each one I list what we assumed would work, what actually drove the numbers, and what we learned before moving to the next step.

where time goes (current 3x3 h24 model)

At 39 inputs and 24 hidden neurons, the full neural step breaks down roughly as:

• pack_player_features(): 27 voxel lookups + 12 scalar writes.
• Layer 1 (39 -> 24 int8): 936 int8 multiply-adds with per-row scale. Dominates forward pass (~0.76 us).
• Layer 2 (24 -> 7 int8): 168 multiply-adds. Minor.
• State apply: write pos delta, velocity, grounded flag. No collision pass.

AVX2 FMA lives only in neural.c (not whole-program -march=native); global AVX flags caused stack misalignment crashes when calling raylib from main.

Compared to analytic (~0.08 us), the network still does hundreds of multiply-adds per tick. We removed the speed-ladder models (h12/h16/h32) from disk; they were faster on paper but worse on rollout.

benchmarks

lessons

one-step accuracy lies

Every checkpoint looked fine on held-out one-step validation (98-99% grounded). Early writeups sometimes reported those numbers as if they were long-run rollout scores. They are not. Rollout benchmarks (50 episodes x 300 steps, analytic vs neural in parallel) show ~89% grounded agreement for the default h24 model. Errors compound: position drifts, the 3x3 patch sees the wrong voxels, and the player phases through blocks (~2.4% of airborne steps in bench). Analytic physics has zero such overlaps.

why old builds felt more stable

Crop-trained weights from the first 3x3 sweep are gone; we re-recorded native 3x3 data and retrained. One-step metrics stayed similar but rollout changed. There is no analytic collision fallback (removed on purpose). If an older build felt smoother, it may have been different weights, not a magic architecture change. The current default is the best rollout model we still have on disk.

patch size mattered more than hidden width

Going from 741 to 355 to 39 inputs gave larger wins than any matmul kernel tweak. The 9x9 assumption was safe but expensive. 3x3 was the risky bet that paid off for this specific world generator.

implementation details compound

Release flags, fused normalization, single-pass feature packing, and int8 export each gave meaningful gains alone. Stacked together they turned a research prototype into something that runs at ~0.95 us (h24 default) without changing the fundamental fact that matmul is slower than analytic collision code.

bigger models do not fix rollout

h48, h64, and 48x24 all beat h24 on one-step metrics but score worse on rollout and tunnel counts. A 5x5 patch experiment did not beat h24 on 3x3 either. More neurons or context within this MLP formulation does not solve open-loop drift. Next levers: multi-step training, or a minimal post-neural depenetration pass (nudge out of solids without full analytic physics).

analytic will always win on raw speed

The goal was never to beat analytic. It was to get neural close enough that the swap is a viable design option. At ~12x and well under 0.01% of a 60 FPS frame budget, we are there for latency. Rollout fidelity is the open problem.

reproduce

cd build && cmake .. -DCMAKE_BUILD_TYPE=Release && cmake --build .
./voxel_parkour.exe --record 500000 --out ../data/train.bin

python tools/train.py --sweep-stable --epochs 40   # h48, h64, h48x24 on 3x3 data

./voxel_parkour.exe --bench ../models/model_q8_h24_p3.bin
./voxel_parkour.exe

Tab toggles analytic and neural physics mid-game. N reloads the model. Game loads model_q8_h24_p3.bin first, then falls back through h48x24, h64, and h48. Training takes ~13 min for the stable sweep on CPU; --log-every 5 prints ETA during training.

what is next

Rollout is ~89% on the default model, not perfect. Phasing still happens in play despite good one-step numbers. Likely next experiments: multi-step training loss (teach stable trajectories, not single frames), or a lightweight depenetration clamp after the neural step (not full analytic fallback). Residual learning (predict a delta on top of analytic) is another path.