2016 ML thoughts


(Old notes from 2016 that I stumbled upon).

  • Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
  • Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
  • Cloze deletion should be network’s default task
  • Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
  • I wonder if all this will finally teach us complex systems
  • If it picks up language, we could just try talking to it.
  • I bet people will say it’s ”not real intelligence”. Whatever bro, it’s more coherent than you
  • All these architectures don’t seem to matter that much
  • If it ever learns reasoning I feel like we’re screwed
  • Normalization seems really important but batch norm is weird
  • Statistical learning theory doesn’t seem helpful
  • Why can a network be compressed so much?
  • Is parity the hardest function to learn

Related Posts

Minimal Surfaces

November 2, 2023

NTK reparametrization

Kate from Vancouver, please email me

ChatGPT Session: Emotions, Etymology, Hyperfiniteness

Some ChatGPT Sessions

My biggest takeaway from Redwood Research REMIX

finite, actual infinity, potential infinity

Actions and Flows

PSA: reward is part of the habit loop too