Policy ManifoldsInteractive Reinforcement Learning π(a|s) lab
Loading checkpoint
Reproduce my work 🔗
Interactive Reinforcement Learning π(a|s) lab

Change an edge in the neural network.
See how the RL agent's behavior changes.

CartPole-v1
TRAINING ALGORITHMPPO — Proximal Policy Optimization
Ready
Episode 01
Live policyOriginal policy
LIVE REWARD0 / 500
ORIGINAL REWARD0 / 500
POLE ANGLE0.0°

Balance over time

LiveOriginal±12° limit
Pole angle · degrees500 steps / episode

Next action

Original agrees
← Push left 50%50% Push right →

Probabilities at the live state. The higher one determines the next push.

Drag a control point while the policy runs.
UNDERSTAND THE POLICY

What is Policy Manifolds?

An interactive reinforcement learning lab where you can inspect a trained policy, reshape its learned functions, and watch the agent respond.

π(a|s) describes an agent’s policy: how it chooses an action a given its current state s.

In a regular MLP

Each edge scales an input by a learned weight. Neurons combine their inputs, then apply nonlinear activation functions such as ReLU or sigmoid.

In a B-spline KAN

Each edge applies a learned function. B-spline coefficients shape that function, and each node sums its incoming edge outputs. Here, an edge combines a B-spline with a weighted SiLU term.

Policy Manifolds trains KAN policies with reinforcement learning, then exposes their learned curves for inspection and editing.

  1. Compact neural networks. Explore how few parameters a policy needs to perform well. The app shows each saved model’s size and evaluation results; compactness depends on the task.
  2. Control through curve editing. Move a spline control point and the next action uses the edited function. Observe changes in behavior immediately, without retraining.
  3. Mechanistic interpretability. Inspect how inputs flow through edge functions, then intervene on a curve to test its role in a decision. Compare with the original policy from the same starting state.

A visible curve is a starting point for understanding, not a complete explanation. Behavior depends on the whole network and the environment; paired episode comparisons help test an interpretation.

Architecture background: KAN: Kolmogorov–Arnold Networks ↗
A HANDS-ON EXPERIMENT

Reshape a decision.

  1. Choose an environment with a saved policy from the dropdown in the top bar, then press Run policy.
  2. Select an edge in the network or one of the curve cards.
  3. Drag a diamond up or down. You are changing a real spline coefficient; the next action uses the new curve.
  4. Watch the simulation, next action, and progress trace. The dashed ghost keeps the original policy.
  5. Use Restart for the same starting state, Undo to reverse an edit, or Restore trained policy to reset everything.

Try a middle control point on an angle or velocity edge. CartPole chooses between two pushes; Pendulum applies continuous torque. Acrobot and MountainCar choose among three actions. Use “Compare 20 episodes” for a broader comparison.

The checkpoint’s actor runs in your browser. Classic Control physics runs in the browser; Box2D and MuJoCo use the local Python server. Larger networks have an input-group selector, and every actuator has its own control indicator. Saved evaluation badges distinguish target-passing, below-target, and incomplete runs. Each environment loads its own saved actor; Pendulum normalizes angular velocity by 8 and maps its action head through tanh to ±2 N·m. InvertedPendulum uses its saved observation normalization and continuous cart controls from −3 to +3. The action head stays fixed. Diamonds edit B-spline coefficients at fixed x positions; these are not Bézier tangent handles.