Stupid Simple Algorithms: DAgger


My hatred of the fetish for acronyms in academia reminds me of Amberley Vail’s rants against Imperial Guard jargon in the Ciaphas Cain novels.

They could have just called it Dataset Aggregation. It’s not that long of a phrase and it has nothing to do with the weapon anyway.

I’m assuming you’re familiar with the point of imitation learning.

Our expert policy is , our learner is , and the set of all observations and all actions is .

DAgger is an algorithm which boils down to this simple idea: Try things for yourself, but use your teacher to criticize your actions as long as you have their expertise available.

That’s basically it. Here’s a sketch of the whole process, assuming you have some training data to start with.

  1. Train with .
  2. Execute to get observations and actions .
  3. Use to tell you what you should have done. AKA, replace with where .
  4. Train with this modified data.
  5. Repeat till is good enough or your funding runs out or something.

The key thing is that the actions actually taken, , are picked by the learner, but the expert/teacher tells you what you should have done, with .

This avoids the issue of the expert and learner policies inducing different trajectories over the state space by aggregating the data they generate. Rather than trying to make the polices exactly the same, you make their data the same (in the limit).

It even works if you sometimes let any other policy (usually the expert) pick the action. It works best if that other policy is better than your own. This is well suited to something like autonomous driving, where you don’t want to hit a tree and you’d still like your training to work.

Related Posts

e-ink reminisces

Edwin Edwards

Middle School by Bo Burnham

How to Disable Disqus Ads on your Blog

Derivation of Reservoir Sampling

Fun with Python Iterators: Linked Lists Made Easy

Notes for November 11, 2018

Underrated Vim Option: undofile and undodir

Hot Take on Solo Travel: Starve

Alan Perlis