Shortcut or shelter?

One fork, two routes.

I chose a small environment because its states, transitions, and rewards can be enumerated, and its optimal policy can be computed before training begins.

The agent starts at a fork. EAST enters a short corridor with hazards on both sides. SOUTH begins a longer route protected by a wall and the edge of the grid. The same actuator noise applies everywhere; the southern route is safer because of its geometry, not because its controls are more reliable.

Research question. How reliable must the world be before the shortcut is worth taking?

The purpose is not to rank the algorithms. It is to locate a policy change, explain why it occurs, and test whether finite training recovers the predicted boundary.

The central corridor reaches the goal in eight eastward moves. The southern route is four moves longer under perfect control. Every movement costs$-0.06$, reaching the goal adds $+8$, and future rewards are discounted by $\gamma=0.98$.

I keep the geometry fixed and change the consequence law. A recoverable hazard adds $-0.50$ and the episode continues. A lethal hazard adds $-8$, ends the episode, and removes the chance to reach the goal.

The quantity being learned.

A policy $\pi$ maps a state to a distribution over actions. Its return from time $t$ is the discounted sum of future rewards:

The action value is the expected return after one action and the policy that follows:

A boundary in reliability.

Let $s_0$ be the start. Define the start-state action gap

Positive values favor the corridor; negative values favor the shelter. The policy boundary $p^\star$ is the zero crossing.

Because the transition and reward model are known, the optimal values satisfy the Bellman equation and can be solved directly:

The lethal boundary is compressed close to perfect execution because the corridor is not a single gamble. It requires repeated exposed decisions. In a simplified passage requiring $m$ clean movements, a clean run scales as $p^m$. The full maze is solved exactly rather than reduced to that expression, but the compounding mechanism remains.

World noise is not agent exploration.

Reliability $p$ belongs to the environment. An intended action executes correctly with probability $p$; otherwise it slips left or right with probability $(1-p)/2$ each.

Exploration $\epsilon$ belongs to the behavior policy. Here,

Lowering $p$ changes the dynamics. Raising$\epsilon$ changes the continuation policy being valued. The distinction survives even at $p=1$: perfect actuation cannot prevent an exploratory policy from selecting a bad action.

In the lethal maze, persistent $\epsilon=.10$ exploration is costly enough that the exact epsilon-soft solution prefers the shelter even under perfect actuation. Frozen-greedy deployment and continued online exploration are different decision problems.

Three rules, two continuation objectives.

All three methods update an action value by

They differ in the target $Y_t$. Here$d_{t+1}$ is one only for a true terminal transition; the continuation value is then zero.

Under persistent exploration, Q-learning should approach the greedy boundary, while SARSA and Expected SARSA should approach the epsilon-soft boundary. The difference is the continuation value inside the update.

The training unit is an interaction.

Shelter-taking policies produce longer episodes. Equal episode counts would give them more transitions and more updates, so every primary trial receives the same 100,000-interaction budget.

Episodes are capped at 250 steps. Time-limit truncation resets the rollout but does not suppress the bootstrap; true termination does. The exact-model reference is the corresponding discounted infinite-horizon MDP, so truncation frequency is one of the empirical checks rather than an unspoken equivalence.

Numerical reference solutions determined where the reliability grids required resolution. Small pilot runs then calibrated coverage, learning rate, and budget. The full configurations were fixed after that calibration. This is a prespecified full-run design, not a claim of preregistration.

Uniform optimistic initialization is a declared coverage device. The primary learned estimand is the fraction of independent seeds whose final greedy start action is EAST. A tie, NORTH, or WEST remains classified as other; it is not silently converted into one of the expected routes.

Training seeds carry the uncertainty. Evaluation episodes within one trained policy are repeated measurements, not additional replicates.

Predictions before analysis.

1. Q-learning should place its finite-budget route transition near the greedy model boundary.
2. SARSA and Expected SARSA should place theirs nearer the persistent-epsilon model boundary.
3. At a common Q-table, state, transition, and behavior policy, SARSA and Expected SARSA targets agree in conditional expectation; Expected SARSA removes the next-action component of target variance.
4. When exploration anneals to zero, all three methods should move toward the same greedy route where the two exact objectives disagree.

The fourth prediction is the important check. If the methods do not converge toward the common greedy route, the continuation-objective explanation is incomplete. Insufficient optimization, poor coverage, constant-step-size behavior, or the training budget would still need to be considered.

What the evidence can support.

The final result will be conditional on the topology, reward law, discount, exploration rule, initialization, and 100,000-transition budget. It will not establish that SARSA is universally safer, that Q-learning is inherently reckless, or that one method is generally superior.

The EAST/SOUTH summary is also restricted to the plotted high-reliability domain. At sufficiently poor control, NORTH or WEST can become exact-optimal.