(Old notes from 2016 that I stumbled upon).
- Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
- Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
- Cloze deletion should be network’s default task
- Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
- I wonder if all this will finally teach us complex systems
- If it picks up language, we could just try talking to it.
- I bet people will say it’s ”not real intelligence”. Whatever bro, it’s more coherent than you
- All these architectures don’t seem to matter that much
- If it ever learns reasoning I feel like we’re screwed
- Normalization seems really important but batch norm is weird
- Statistical learning theory doesn’t seem helpful
- Why can a network be compressed so much?
- Is parity the hardest function to learn