In the previous post I referred to a fantastic conversation between Satya Nadella and Dwarkesh Patel (Microsoft’s AGI plan & quantum breakthrough, Feb 19, 2025).
I was moving upstream to find the original source for a quote on AI benchmarks:
…that’s just nonsensical benchmark hacking to me.
For my MEd research I was looking at the relationships between AI, divergent thinking, and creativity. So I was reading a lot of claims by AI companies about the power of LLM models. Scoring new and unbelievable heights on standardized tests.
But it feels foolish, doesn’t it? I mean, there is an answer key somewhere and the worst kind of education is where everyone teaches for the test. Don’t we loose so much in the process of learning if all we do is focus on answering the test? So I’m with him on that one, AI benchmarks are kind of nonsense.
The whole conversation proved to be fascinating. It led me to many other insights and ideas. He is an interesting guy and perhaps I’ve been wrong about Microsoft (since 1994).
He went on to say:
Before I get to what Microsoft’s revenue will look like, there’s only one governor in all of this. This is where we get a little bit ahead of ourselves with all this AGI hype. Remember the developed world, which is what? 2% growth and if you adjust for inflation it’s zero?
So in 2025, as we sit here, I’m not an economist, at least I look at it and say we have a real growth challenge. So, the first thing that we all have to do is, when we say this is like the Industrial Revolution, let’s have that Industrial Revolution type of growth.
That means to me, 10%, 7%, developed world, inflation-adjusted, growing at 5%. That’s the real marker. It can’t just be supply-side.
In fact that’s the thing, a lot of people are writing about it, and I’m glad they are, which is the big winners here are not going to be tech companies. The winners are going to be the broader industry that uses this commodity that, by the way, is abundant. Suddenly productivity goes up and the economy is growing at a faster rate. When that happens, we’ll be fine as an industry.
But that’s to me the moment. Us self-claiming some AGI milestone, that’s just nonsensical benchmark hacking to me. The real benchmark is: the world growing at 10%. (0:15:18 – World economy growing by 10%).
Wow. Um… let’s have it.