Trevor Chow

Two weeks ago, OpenAI released the o1 series. This publicly marks the start of the “search” paradigm in modern ML, just as ChatGPT’s launch in 2022 marked the arrival of the “learning” paradigm.  With this new paradigm, you should expect progress in ML performance over the next two years

Parameters and data. These are the two ingredients of training ML models. The total amount of computation (“compute”) you need to do to train a model is proportional to the number of parameters multiplied by the amount of data (measured in “tokens”). Four years ago, it was well-known that if