Unit 01 of 02
Reinforcement Learning
Measuring an agent that learns the result only after the action and collects its own data: in the field visit setup, the return of the random policy against that of the learning policy and the states never seen; the change in return when a history field is added to the assumption that the transition depends only on the current state; the regret that accumulates as the exploration rate is swept and the action the always-exploiting policy never tries; the cells never updated in a table learned with the temporal difference update; how correlated consecutive samples spoil training and the difference experience replay makes; how many distinct directions the gradient estimate gives under the same budget and how the baseline lowers this; the gain brought by the advantage estimate and where the actor goes when the critic is bad.
Published Updated
Lessons
- 01 Agent, Environment and Reward Reading three numbers in a setup where the decision produces its own data: in the field visit setup, the policy that never dispatches the crew gets a return of 0.000, random action, the baseline, gets 10.432, and the rule that goes to the subscriber with the largest delay gets 20.621. The third number is the collected data: the random policy sees 234 of the 243 states and 1,251 of the 1,458 state-action pairs, while the rule, which earns the better return, sees only 103 states and 103 pairs. In the same period the six actions would give rewards of -2, -2, 1, -2, 1 and 0, and the agent learns only one of them; this is what takes the place of the label in supervised learning. 14 min read
- 02 Decision Process Formalization Writing down the five-tuple of state, action, transition, reward and discount factor, and measuring its defining condition in the setup: in the same state-action cell the next state spreads over four distinct values with probabilities 0.4307, 0.2239, 0.1770 and 0.1046, and the reward over -2, 1 and 7 with 0.0516, 0.2347 and 0.7137. The memoryless-transition test shows that the condition does not hold exactly: in the most frequent cell, for the same state and the same action, the mean reward is 4.518 when the delay is 3 and 5.944 when the delay is 4 or more; the difference is 1.425 and positive in five cells out of five. The same hand-written rule gets a return of 20.621 under the 243-state definition, 23.148 under the 1,215-state definition with a history field added, and 29.053 under the upper reference that sees the hidden level. 13 min read
- 03 The Exploration and Exploitation Trade-Off Counting the price paid when learning requires trying and winning requires repeating what is known: the agent that never explores, starting from a pessimistic initial value, never tries five of the six actions, sees 95 of the 243 states and gets a return of -16.492, while its regret is exactly 0.0. When the exploration rate is swept from 0.00 to 1.00 the return is -16.492, 16.716, 22.210, 23.606, 25.659, 25.997 and 23.000; the accumulated regret is 0.0, -76.0, 402.0, 1,210.0, 3,545.0, 8,358.0 and 33,811.0. A return 0.338 below the best return can be had at a rate of 0.20 with a regret of 3,545. When the initial value of the estimates is moved from pessimistic to optimistic, the agent that never explores rises to 26.212 and its regret stays at 0.0. 13 min read
- 04 Value Functions and Action Value Learning value only from lived quadruples, without knowing the environment's probabilities, and counting the empty places of the table: when the temporal difference update is swept 40 times over a 9,000-step trace, the mean value of the starting states rises from 12.632 to 21.562, and its gap to the measured 30-period return of 20.621 sits on the 0.0424 tail, the thirtieth power of the discount factor. State values spread between 29.060 and 16.075. Of the 1,458 cells of the action value table fed by its own data only 103 are updated, 1,355 stay untouched, and the policy read from the table gives exactly the rule that produced it, 20.621. As exploration is added to the behavior, the filled cells rise from 103 to 1,237, but the thin cells with fewer than five samples also rise from 47 to 468, and the return does not move in one direction. 10 min read
- 05 Function Approximation and Experience Replay Measuring, in numbers, a small network that replaces the table and the experience replay: of the 1,458 cells formed by 243 states and 6 actions, only 780 are updated after 9,000 transitions, 37 states are never seen, and only 60 states have all six actions tried. A sequential batch of six comes from 1.16 distinct episodes on average, a batch of six drawn from the buffer from 5.50. The baselines are 0.000 and 10.432, the rule policy 20.621, the table 24.495. The network with 126 parameters gives 23.211 and 23.071 with online single steps, 22.259 and 20.146 with sequential batches of six, 26.233 and 25.302 with the replay buffer; the run band falls from 2.113 to 0.930. 9 min read
- 06 Policy Gradient Measuring the cost of parameterizing the policy directly: with the same policy and the same budget of 120 episodes, the 120 update directions read from single episodes make a mean angle of 86.1 degrees and the agreement stays at 0.290; with six estimates of twenty episodes each, the angle rises to 44.0 degrees and the agreement to 0.878. When the baseline built from the step-by-step mean return is subtracted, the same numbers become 81.2 degrees with 0.389 and 39.8 degrees with 0.892; two episodes with baseline subtraction give the agreement of four raw episodes. In training with 240 episodes, the two-episode estimate gives 15.840 and 19.962 with raw return and 22.866 and 22.190 with baseline subtraction; the run band falls from 4.122 to 0.675. 13 min read
- 07 Actor–Critic Methods Measuring the joint training of the component that chooses and the component that evaluates, and how they spoil each other: under the same budget of 240 episodes, with a step of 0.01 the raw return gives 15.840 and 19.962, the fixed baseline 22.866 and 22.190, the advantage estimate of the learned critic 21.154 and 21.100. When the learning step is raised to 0.06, the two methods that rely on the full return break down — one run of the fixed baseline falls to the never-dispatch baseline with 0.029, its band 15.953 — while the advantage gives 23.401 and 23.755 and keeps the band at 0.354. When the learning step of the critic is raised to 0.50, the values diverge to the order of 10 to the power of 102, the actor piles onto a single action, four of the five subscribers are never visited and the return falls to -18.772. 11 min read