Stupid Simple Algorithms: DAgger

My hatred of the fetish for acronyms in academia reminds me of Amberley Vail’s rants against Imperial Guard jargon in the Ciaphas Cain novels.

They could have just called it Dataset Aggregation. It’s not that long of a phrase and it has nothing to do with the weapon anyway.

I’m assuming you’re familiar with the point of imitation learning.

Our expert policy is , our learner is , and the set of all observations and all actions is .

DAgger is an algorithm which boils down to this simple idea: Try things for yourself, but use your teacher to criticize your actions as long as you have their expertise available.

That’s basically it. Here’s a sketch of the whole process, assuming you have some training data to start with.

  1. Train with .
  2. Execute to get observations and actions .
  3. Use to tell you what you should have done. AKA, replace with where .
  4. Train with this modified data.
  5. Repeat till is good enough or your funding runs out or something.

The key thing is that the actions actually taken, , are picked by the learner, but the expert/teacher tells you what you should have done, with .

This avoids the issue of the expert and learner policies inducing different trajectories over the state space by aggregating the data they generate. Rather than trying to make the polices exactly the same, you make their data the same (in the limit).

It even works if you sometimes let any other policy (usually the expert) pick the action. It works best if that other policy is better than your own. This is well suited to something like autonomous driving, where you don’t want to hit a tree and you’d still like your training to work.

Related Posts

How I feel about ebooks

List of places where the US has been involved in regime change, with multiplicity

Accuracy vs Precision

Handy command line benchmarking tool

Stan Rogers

Ultimate Hot Couch Guy

Quote on Java Generics

The Programmer Tendency

Figure out undocumented JSON with gron

Mental Model of Dental Hygiene