Collections

People have a misconception that it takes a lot of exceptional people to make frontier models. You need a handful of experienced people, but mostly you need a lot of motivated and talented people… This technology is not that old. The people who have experience like myself only have a few years of experience working on this. A lot of people can build that experience very quickly.

It is good for the world if there are thousands of models, by different providers. Competition breeds innovation, and specialization is better than one size fits all. Different users want models at different sizes and different price points. Specifically for India, we have so many languages and cultures, and our own education, legal and political systems, our own public policy. So we want models that can be customized for all these things, and that is why we need to build these models here.

With scaling laws, people realized that scaling pretraining is the thing to do. The big breakthrough of pretraining is the realization that this recipe is good: if you mix some data with some compute into a neural net of a certain size, you will get results, and you know you will get something better if you scale the recipe up.

At some point though, pretraining will run out of data. The data is very finite. So then we either do souped up pretraining or some kind of RL, and in some sense we are back to the age of research.

From 2012 to 2020, it was the age of research. From 2020 to 2025, it was the age of scaling. But now the scale is so big that the belief that things would be significantly different if it were even 100x bigger might not be true. So we are back to the age of research again.

Current LLM agents can do really well and are incredibly capable when they are given a clear and detailed specification. With access to these kinds of agents, the scarce skill then becomes having incredibly good taste in what you ask them to work on.

A researcher can have all the tools and techniques, but often most of the battle is what problem you are going to spend your time on. That high level wisdom of what to work on is incredibly important, and I think models are not necessarily going to be good at it. We are going to have people steering a lot of AI-assisted computation in order to accomplish great things more quickly, but the essence of what you want the models to do is the key thing to focus on.

Building taste is difficult. Working on a lot of problems in the past teaches you what kind of problems might be interesting in future, and what might just be possible by cobbling together previous approaches. Another way is to write down a bunch of things you think might be important in the near future, pick one of them to work on, and after a certain amount of time go back and evaluate which of the others turned out to be important and worked for other people. That gives you a lot more samples to build taste from. Finally, do crazy thought experiments, and don’t take as given the things that most people seem to take as given.

You insist that there is something a machine cannot do. If you tell me precisely what a machine cannot do, then I can always make a machine which will just do that.

You can outsource your thinking but you cannot outsource your understanding.

When you grow up you tend to get told the world is the way it is and your life is just to live your life inside the world. Try not to bash into the walls too much, try to have a nice family life, have fun, save a little money. But that is a very limited life, life can be much broader. Once you discover one simple fact, and that is everything you call life was made up by people that were no smarter than you. And you can change it, you can influence it. You can build your own things that other people can use. And the minute that you understand that you can poke life, and if you push in something will actually pop out the other side, that you can change and mold it.

The most important thing is to shake off this erroneous notion that life is there and you are just going to live in it versus embrace it, change it, make your mark upon it. However you learn it, once you learn, you’ll want to change life and make it better because it is kind of messed up in a lot of ways. Once you learn that you will never be the same again.

In future people won’t necessarily be having shitty jobs or nothing to do. Instead they will be handling and guiding the Robots and AI. If we don’t get ASI, if AI gets you to expert level but doesn’t go beyond and still needs human taste, guidance and creativity, humans are still the motivated things in the society. People become like Pokémon trainers or robot/AI handlers.

That is what we are seeing today. People who are using AI today are more productive than ever. The only way you are being displaced by AI if you are refusing to use it.

There will be 3 types of people in the future:

The Purists - reject AI and become new-age hippies

The Automatons - outsource every decision and become machines

The New Renaissance Man - maintains humanity and leverages tech to do what used to be impossible

The problem was you can’t just ask Aristotle questions. And I think as we look to the next 50-100yrs, if we really can come up with these machines that can capture an underlying spirit… maybe someday after a person’s dead and gone, we can ask this machine, hey what would Aristotle have said?

And maybe we won’t get the right answer. But maybe we will. And that’s really exciting to me. And that’s why I’m doing what I’m doing.

Computer Science is no more about computers than astronomy is about telescopes

I visualize a time when we will be to robots what dogs are to humans. And I am rooting for the machines.

The first principle is that you must not fool yourself—and you are the easiest person to fool.

The book of nature is written in the language of mathematics.

The way a computer and a keyboard are to a cat, the intricacies of the universe are to a human. We will probably never understand it. But it is good once in a while to discover something new.

The 80% Fallacy: When you start building using AI, you’re likely to get pretty good results very quickly. However, the journey from 80% to 100% is far more challenging than the journey from 0% to 80%. It’s a reality that catches many beginners off guard, leading to wasted resources, missed deadlines, and failed projects.

The Exponential Difficulty of Completion means that as AI projects advance, challenges and complexities multiply. Minor issues in the early stages can suddenly become major roadblocks.

[This is] due to two main factors: Technical Complexity (primarily for Classical AI) and Unpredictable Outputs (especially for Generative AI).

Singularity refers to a future point in time where artificial intelligence (AI) could potentially surpass human intelligence. This occurrence would arise from the exponential self-improvement of AI, leading to an unpredictable and rapid change in the technological landscape.

Jagged intelligence describes the strange, unintuitive fact that state-of-the-art LLMs can perform extremely impressive tasks, like solving a complex math problem, while simultaneously struggling with some very dumb ones.

Shoggoths are formless, powerful, and not entirely controllable creatures from Lovecraft’s stories. Large Language Models are analogous: they are almost an alien technology, and we don’t entirely understand how or why they do what they do. Since we don’t understand them, we can’t be sure we can control them. The cheerful smiley face stuck on top of the shoggoth represents what we see after reinforcement learning from human feedback (RLHF): a polite, aligned chatbot. Underneath, it is still a Shoggoth.

AI accelerates the generalist and polymath - you can become a one-human business. Content on social media to attract an audience around your life’s work, code to build the product, funnel, and digital real estate, and domain specific knowledge that makes what you do valuable over the next person.

I am not sure inference will essentially ever be free. Because of Jevons paradox and things like that, I think all of us will just end up using whatever we can get our hands on. We can imagine millions of swarms of agents working together on things, and this is one way to use inference. Or you could imagine smaller groups of agents thinking in multiple directions and then ensembling that. All of that will end up using any inference that is available.

One day it could be cost zero if we solved fusion, superconductors or optimal batteries with material science, and energy costs came down to zero. But there will still be some bottleneck, like the physical creation of chips, at least for the next few decades. If that is the case, then there will be rationing on the inference side and we will have to use it efficiently.

Young people should focus on using the AI tools that we have built, and master even those things that we missed because we were busy building the models.

Morris Chang, 1955

Vasili Arkhipov, 1962