Estimands
Each experiment names the quantity it is trying to learn about before choosing a plot. Route choice, action-value error, regret, and held-out return answer different questions.
Methods
How the experiments are designed, run, evaluated, and reported.
Exact model references
For known finite transition models, dynamic programming establishes the reference policy and identifies the parameter region where learning deserves close inspection.
Seeds and uncertainty
Episodes within one trained policy are repeated measurements. Uncertainty is carried by independently trained seeds, with matched seed panels used for method contrasts.
Interaction budgets
Policies can induce trajectories of different lengths. Counting transitions rather than episodes keeps the amount of experience comparable.
Held-out evaluation
Exploratory training trajectories and update-free held-out evaluations are recorded separately. Greedy deployment and continued exploratory behavior are separate evaluation conditions.
Exceptions and failures
Nonmonotone curves, ties, censored boundaries, instability, failed runs, and protocol changes are included in the reported record.
Implementation.
Environment definitions, agents, exact solvers, experiment protocols, tests, and versioned configurations live in the project repository.
Open the source