Transformer gives you function $$F: [X] \to [Y]$$. Anything that can be totally ordered can be learned. So it’s pretty general.

To me, this paper is more evidence for the transformer as a universal architecture. Some other (famous) examples:

• AlphaFold
• GPT
• BERT

## The Idea

Take an image (and its target image in training), cut it into patches, embed each patch, then use transformer on that. It’s that simple.

I wonder if different sequentialization strategies would work better or worse (rasterize up/down instead of left/right)? Would doing it randomly give noticeably different performance?

## Bitter Lesson

We find that large scale training trumps inductive bias.

I gotta get more compute.