Shortcut
or shelter?
A policy boundary in a noisy maze—and a test of whether Q-learning, SARSA, and Expected SARSA learn to value the same journey.
THE QUESTION
One fork, two defensible routes.
The agent starts at a choice. EAST enters a short exposed corridor. SOUTH begins a longer route sheltered by a wall and the edge of the grid. An execution error can be harmless on the protected route and disastrous in the corridor.
As intended-action reliability p increases, there should be a point where speed becomes worth the risk. This study asks where that point lies, how hazard consequences move it, and whether different temporal-difference backups approach different boundaries while exploration continues.
Figure 1. The same actuator noise applies everywhere. Geometry—not privileged control—makes the southern route safer.
TWO RANDOMNESSES
Noise in the world is not exploration by the agent.
The probability that the environment executes the movement the agent intended.
The probability mass the behavior policy assigns to uniformly sampled actions.
Lowering p changes the environment. Increasing ε changes the policy being followed. A route can therefore be optimal for frozen-greedy deployment but unattractive when exploratory actions continue after training.
EXACT FIRST
DETERMINISTIC MODEL CALCULATION · NOT A LEARNED RESULT
The environment tells us where to look.
At the starting state, define the action gapΔ(p) = Q(s₀, EAST) − Q(s₀, SOUTH). Its zero crossing is the route boundary. Because the transition and reward model are known, dynamic programming gives exact predictions before a single agent is trained.
Recoverable hazards
Greedyp* ≈ 0.806corridor above the boundaryRecoverable hazards
ε = .10 softp* ≈ 0.848exploration moves the boundaryLethal hazards
Greedyp* ≈ 0.989a narrow transitionLethal hazards
ε = .10 softNo crossingshelter through p = 1THE BACKUPS
Same maze. Different continuation values.
This is a controlled comparison of backup rules, not a universal algorithm leaderboard. Hyperparameters and interaction budgets are matched.
Q-learning
Bootstraps toward the best next action. Its target corresponds to greedy control.
SARSA
Bootstraps from the next action actually sampled, including the consequences of continued exploration.
Expected SARSA
Averages over the behavior policy. It shares SARSA’s expected target while removing next-action sampling noise.
THE PROTOCOL
Count interactions, not attempts.
Shelter episodes are longer, so equal episode counts would grant some policies more updates than others. Every primary trial instead receives the same 100,000-transition budget.
- Training unit
- 100,000 interactions
- Independent unit
- Training seed
- Primary endpoint
- Final greedy fork action
- Checkpoints
- Every 2,500 interactions
- Deployment view
- Frozen greedy
- Online view
- Continuing behavior
EMPIRICAL RESULTS
LOCKED UNTIL ARTIFACT VALIDATION
The axes can wait.
Data collection is complete: all 1,992 planned trials reached succeeded manifest states, with no recorded failures. This section will be populated only after the integrity, calibration, and late-training stability checks pass.
PLANNED EVIDENCE
- Seed-level corridor-selection curves
- Paired boundary estimates and uncertainty
- Boundary movement over training
- Frozen versus continuing-behavior returns
- Exact deployment regret and hazard exposure
- Policy atlas and declared-seed rollouts
INTERPRETATION
What the result will—and will not—mean.
- One environment, one greedy optimum. An algorithm does not get its own ground truth.
- Persistent exploration changes the objective. Greedy and epsilon-soft boundaries can legitimately differ.
- Seeds, not episodes, carry uncertainty. Repeated evaluation episodes are measurements within a trained policy.
- Finite training remains visible. Instability, ties, and off-route actions are evidence, not cleanup tasks.