Choosing an RL Algorithm

There is no generally best reinforcement-learning algorithm. First ask what decisions the agent makes, how experience is obtained, and what failure costs. A method that looks strong on a cheap simulator may be unsuitable when each real action is expensive or unsafe.

Choose a family before tuning a library: small discrete problem and known state, high-dimensional observations, continuous control, fixed logged data, or a useful dynamics model. Then compare candidates under the same environment, observation and reward definitions, interaction budget, compute budget, and evaluation protocol.

Use this table as a shortlist, not an automatic winner. The right branch depends on data quality and engineering constraints.

Safety is a constraint on the experiment, not a property inferred from an algorithm name. If a bad exploratory action is unacceptable, define a safe simulator, action shield, constrained policy, or offline evaluation plan before running online RL. A reward score alone does not establish safety.

Start with the discounted return $G_t=\sum_{k=0}^{\infty}\gamma^k r_{t+k+1}$. Value-based methods estimate expected return for an action; policy-gradient methods adjust action probabilities to improve expected return. An off-policy method can train from data generated by another behavior policy; an on-policy method such as PPO uses fresh samples from a recent policy for its update.

For Q-learning, a one-step target is $r+\gamma\max_{a'}Q(s',a')$ for a nonterminal transition. For PPO, the update uses an advantage-weighted likelihood ratio clipped in its surrogate objective. SAC uses replay with an entropy-regularized actor-critic objective, often improving interaction efficiency on the continuous-control tasks it was designed for. These are different optimization signals; they do not form an accuracy ranking.

Offline data add a coverage problem. A learned policy may choose actions absent from the log, while bootstrapped value estimates can assign those actions undeserved high value. CQL addresses this by learning conservative values, but its assumptions and evaluation still matter. Model-based methods trade more computation for simulated experience and must control model bias.

Two budgets must stay separate: environment steps measure interaction cost; wall-clock time includes simulation, model training, data transfer, and evaluation. Compare performance against both axes, across seeds, with the same termination and truncation semantics.

Write the experiment contract before running candidates. This standard-library sketch records the metrics that let a reviewer compare runs; the values come from actual run logs, never from the shortlist table.

1from statistics import median
2
3
4def summarize_run(env_steps, eval_returns, safety_violations, wall_seconds):
5    assert env_steps > 0 and wall_seconds > 0
6    assert eval_returns, "evaluate complete episodes"
7    assert safety_violations >= 0
8    return {
9        "env_steps": env_steps,
10        "median_eval_return": median(eval_returns),
11        "eval_episodes": len(eval_returns),
12        "safety_violations": safety_violations,
13        "wall_seconds": wall_seconds,
14    }

For each candidate, fix action and observation spaces, reward and constraint definitions, evaluation episodes, random seeds, and stop budget. Log behavior-policy coverage for offline data. Inspect learning curves, uncertainty across runs, and the worst safety outcomes. A single best seed or one final score is not enough to select a deployable policy.

1. Data: Why can PPO be a poor fit for expensive real-world interaction? (It needs fresh rollouts for updates; wall-clock throughput cannot create free environment data.)
2. Actions: Why test DQN for discrete actions and SAC for continuous control? (Those are the settings each method was designed around; adaptation is possible but needs a reason.)
3. Offline: Why not simply train standard Q-learning on a fixed log? (Bootstrapping may overvalue actions the dataset rarely or never contains.)
4. Safety: Does a high average return show that exploration was safe? (No. Measure violations and tail failures during both training and evaluation.)
5. Systems: What must be held fixed in a fair algorithm comparison? (Environment and reward definitions, interaction and compute budgets, seeds, eval protocol, and termination semantics.)

Choose candidate families for three cases: a tiny gridworld, online continuous control with expensive samples, and a fixed robot-operation log where new actions cannot be tried. For each, state one plausible method, one reason it fits, one serious failure mode, and the evidence you would require before use. Compare environment steps separately from wall-clock time.

Sources and further reading

• Mnih et al., Human-level Control through Deep Reinforcement Learning (2015): DQN on high-dimensional observations and discrete game actions.
• Schulman et al., Proximal Policy Optimization Algorithms (2017): on-policy clipped surrogate.
• Haarnoja et al., Soft Actor-Critic (2018): off-policy continuous-control actor-critic method.
• Kumar et al., Conservative Q-Learning for Offline Reinforcement Learning (2020): fixed-data distribution-shift problem and conservative values.
• Janner et al., When to Trust Your Model (2019): model-generated experience versus model bias.