“A lot of people want local inference. And in some ways, they assume that the problem is solved now because we’re going to have open source models, we’re going to be able to run them on our computers, and therefore we’re not going to need all the data center stuff.
But from a global allocation perspective, it’s actually pretty terrible that everyone has a bunch of the world’s limited supply of DRAM on their own computer and using it five percent of the time.
You want maximum cloud utilization if you’re trying to deal with this very supply-constrained environment. And in some way the intuition of everyone, of many people, is the opposite of that.”
