← All work

Nexto Bridge

A reinforcement learning policy driving a live Rocket League client at frame rate

Role
Sole developer of the bridge, protocol, safety layer and test suite, built in a fork of the Nexto repository. The policy is Nexto’s published TorchScript checkpoint - I did not train it, and the training code carried in that history is not mine either.
Period
2021 - 2026
Status
Working
Stack
Python · PyTorch · TorchScript · C++ · Binary IPC · NumPy
250 ms
watchdog back to manual
0
online matches it can enter
2
processes, one socket pair, fixed layout

A neural network policy takes over the player’s own car in a running Rocket League session: a BakkesMod plugin inside the game process streams state over localhost to a Python inference loop and applies the returned controller input, with the handover gated so it can never reach an online match.

01The problem

Reinforcement learning agents for Rocket League normally run through RLBot: a dedicated framework, a bot slot, a match built for them. That is a laboratory. The interesting version is the agent driving the same car a human was driving a second ago, in a session the human started, with the camera, name, boost and scoreline untouched.

Doing that means reaching into a running game process, reading its state fast enough to be useful, and writing controller input back - without ever letting it touch a public online match.

02What I built

A C++ BakkesMod plugin hooked into Rocket League’s state and input path, talking to a Python process over two localhost sockets with a fixed binary layout - one direction for game state, one for controller output.

A Python inference loop that loads the TorchScript policy, pins torch to a single thread, builds the observation tensors and samples an action every tick, with no allocation surprises in the hot path.

A handover triggered by holding both stick buttons for 0.4 seconds, switching between MANUAL and agent control. In MANUAL the plugin does not touch the game’s input at all.

03The safety layer

The agent is only allowed to act when Rocket League itself confirms free play, custom training, an offline match or LAN. If the client reports a public online match, or fails to confirm either offline or LAN, control never transfers.

Beyond that: localhost binding only, the locally reported car rather than a player name, rejection of stale packets and wrong sessions, rejection of NaN and infinity in the policy output, and a 250 ms watchdog that hands control back to the game if a fresh command does not arrive.

The control loop between the game and the policyA plugin inside the game process reads game state and sends it over a localhost socket to a Python process, which runs a TorchScript policy and returns controller input on a second socket. Two guards sit on that path: control only transfers when the game confirms an offline or LAN session, and a 250 millisecond watchdog returns control to the player if a fresh command does not arrive.GAME PROCESSRocket Leaguestate · native inputBakkesMod pluginC++ · reads and writesPYTHON PROCESSInference loopTorchScript · one threadObservation buildno allocation in the hot pathgame state · fixed binary layoutcontroller inputtwo localhost socketsSession gateoffline or LAN confirmed by the game,or control never transfers250 ms watchdogno fresh command, stale packet,or NaN — native input resumes
Both guards resolve the same way: the game’s own input handling. Every failure in this system is meant to look like nothing happened, rather than like a car that stopped responding.

04Decisions worth explaining

A fixed binary layout, verified from both sides

The two halves of this system are written in different languages and compiled by different toolchains. A struct layout mismatch would surface as an agent that drives into a wall for reasons no log explains. The repository carries a C++ test that asserts the protocol layout and Python tests that assert the same thing from the other side.

Failing to manual, never to nothing

Every failure mode in the bridge - a dead Python process, a stalled socket, a malformed packet, a non-finite tensor - resolves to the game’s native input handling rather than to a car that stops responding. Degradation should be invisible, not catastrophic.

Deleting the weaker models on purpose

An earlier version bundled several bot ports and a controller chord to switch between them. An accidental chord could silently swap the model underneath you. The bundle now contains exactly one policy and the switch is gone: fewer ways to be surprised is worth more than a feature nobody asked for.

Questions about this project?

I am happy to walk through the architecture, the parts that did not work, or the code itself.

Email me