Soon decent speed across two Mac Studios with 512GB of RAM.
tito 1 days ago [-]
wow cool! I like watching the new models come out and how they end up crammed in to run on local machines. I learned what mxfp4 is thanks to this latest Kimi release - although it sounds like it means that there's less room for compression in the model compared to others.
whatsThisBtn4 1 days ago [-]
0.3tx per second is decent speed?
And it gets worse with every token.
zozbot234 22 hours ago [-]
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, around 27GB for a full 1Mtok context). What you're seeing in antirez's video is essentially the performance we should expect from a 128GiB system that has to load sparsely routed experts in full from disk because there's no real room for caching them.
192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.
whatsThisBtn4 22 hours ago [-]
Would it even finish over a few weeks?
For AI agents this would take a year
tito 21 hours ago [-]
I read about its format being tough to compress further. What should we expect in terms of creating smaller models to run locally?
searealist 21 hours ago [-]
No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.
dannyw 21 hours ago [-]
The OpenAI and Anthropic batch APIs give you a 50% discount.
This is like 98%.
searealist 20 hours ago [-]
I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
zozbot234 20 hours ago [-]
antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights on its maxed out internal storage (2TB). (This is why the linked OP resorts to streaming the sparse parameters from the network which is incredibly slow.)
searealist 20 hours ago [-]
My answer doesn't change if it is M5s. Where is the math showing a 98% discount over K3 on openrouter. Heck, where is the math showing it is any % cheaper? How much electricity will your M5 sip to hit 1M input and 1M output tokens that would cost $3 + $15 there? I bet it is more expensive on the Mac.
dannyw 20 hours ago [-]
I just mean in terms of incremental inference cost.
If you already committed to your hardware, huge models at single or sub-digit TPS are still useful. LLM-as-judge is a good use case.
If your machine is gonna be idle overnight (and the power efficiency is excellent here), why pay openrouter if it’s not an interactive workload?
searealist 19 hours ago [-]
Because it's probably still more expensive in electricity costs compared to openrouter. Certainly it's not 98% cheaper.
smallerize 23 hours ago [-]
That's not what he said. He said with 2x the hardware it will be faster.
xfour 1 days ago [-]
Cool stuff. Do you have a the hardware and a way to bridge the compute? Or just hopeful?
weikju 20 hours ago [-]
Supposedly RDMA over Thunderbolt 5 can allow this.
api 24 hours ago [-]
I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds?
To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig.
Or could you do the same with custom layout of cheap slower RAM?
I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.
colingauvin 19 hours ago [-]
The problem is you'd have to traverse the PCI-E bus every transfer. Even with DMA it still has to physically get off the drive and onto the card. Even with the mythical PCI-E 6, if you had enough NVMEs to saturate, you have to do two transfers to get it to the inference hardware. And that tops out at 128 GB/s which is roughly the speed of DDR5 but with one extra hop.
Optane would actually be useful in this era. Intel was ahead of their time.
api 10 hours ago [-]
That’s why I think you’d have to build boards with raw flash chips and do your own bus to make it rival fast RAM.
colingauvin 10 hours ago [-]
I'm guessing that there are storage vendors that already do that - maybe like Liqid. But then you've got to traverse the NIC as well.
Azantys 1 days ago [-]
0.01 tk/s is unusable for anything, you would wait a whole day for just 1000 token of output, what is the point of projects like this?
SXX 1 days ago [-]
It is fun.
Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone.
Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200.
SOTA LLMs are efficiently compression of all the knowkedge humanity has built. Having ability to run it at home to extract said knowledge is important no matter the speed.
fp64 19 hours ago [-]
A post-apocalyptic desert landscape. The hero descends into the ruins, now a temple. Crude drawings of times before, people riding dinosaurs and mobile phones with arms and legs swinging a whip at the oppressed. He reaches the altar, a old MacBook, bottlecaps and fidgit spinners glued as a mosaic onto it. In the corner three starved men, riding on power generating bicycles providing some artificial light and supplying the machine with divine electricity. He proceeds and types: "we are starving. How can we get food?". Everybody waits in silence. Hours pass. Days. Slowly, letters emerge on the cracked screen: "you are absolutely right —" the screen flickers and fades. Panic. The link to God severed cynically. The hero has to seek out a new device, into distant lands. Maybe this time it will save them
tito 19 hours ago [-]
“How long?” he said.
“Seven and a half million years,” said Deep Thought.
cj17382 10 hours ago [-]
That's not a great argument for having an llm in an apocalyptic scenario.
People tend to think they are the thriving hero but most won't survive the first year.
I'd much rather have a USB stick of prepper disk. It works on any shitty machine I find, and it's not going to take days for an answer on how to make potash or fix a solar panel.
whatsThisBtn4 1 days ago [-]
Even if somehow all the data centers are gone, it's still uselessly slow.
But also that's a pretty extreme hypothetical. Imagine the polymarket on that.
tito 1 days ago [-]
I like seeing the latest and greatest model crammed into new systems to see how it fares. To deal with the speed, one person on reddit suggested using it in an email interface rather than a chat interface.
magicalhippo 1 days ago [-]
> one person on reddit suggested using it in an email interface rather than a chat interface
Kimi Pen Pal. Bring back lettets and postcards. Do OCR, and use one of those 3D printer-like pen plotters write the model output as a letter.
Challenge would be automating the opening and OCR preparation, and the folding and mailing of the return letter. But given it's done commercially it should be possible.
embedding-shape 1 days ago [-]
Email would indeed be fitting for K3 running on a M1 Mac, as it'd take days/weeks to receive a response, which matches with my real-world emailing experience pretty well.
0xc133 23 hours ago [-]
I had my clanker implement this idea in a standalone Rust server that speaks IMAP and SMTP and proxies your emails to an OpenAI endpoint you configure: https://tangled.org/clee.sh/posthorn
Works in mutt; other MUAs may vary.
tito 21 hours ago [-]
lovely! I'm a fan of asynchronous LLM systems like this. There are times where I prefer slow, complete responses to immediate replies.
For example there's a billboard on 101 for Poppy Bank offering 4% interest and I asked my daily LLM to look into it through a voice note. The next morning, lo and behold it says it's an advertised rate, hard to actually get, and businesses aren't eligible. OK, done. Better than getting the response while I'm driving. And even if I wasn't driving, there's a level of, how do you say, it's easy to drop it the next day when I see it vs getting engrossed in the research.
hugopuybareau 1 days ago [-]
Love the email idea
tito 1 days ago [-]
Having the right type of interface makes a huge difference.
It reminds me of when Willow Garage chose to name their bot the TurtleBot, because if they named it anything else, people would think it was fast and capable. But when they called it Turtle Bot, people just kind of liked it and were satisfied with what it did.
At the level of Kimi 3, I probably can code only about 1,000 good tokens per day, too. (thankfully coding isn't my job)
whatsThisBtn4 1 days ago [-]
I can only guess some post purchase remorse.
Need to justify buying an expensive rig that doesn't do what you expected.
Specifically thinking the people they could do something AI with cpu, and realizing it isn't feasible. Happened at my fortune 20 company. They had to get approvals and ofc it was useless. Plenty people tried to explain, but they were the principle engineer, and out ranked everyone.
"It's not going to work", the topic changed, and we never spoke about it again.
ggm 1 days ago [-]
So you subscribe to the belief we won't in future find mentalism in other galaxies or solar systems which operate on mechanisms we don't understand and think v e r y s l o w w w w w w w l y ?
(note. I am not a believer in AGI)
"useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.
tito 1 days ago [-]
Are there any well thought through stories about what this would look like? For example, I'm thinking about like nutrient flow, decision making, energy input, gravitational force, things like that seem to govern the value and speed of intelligence.
indiv0 24 hours ago [-]
Hard to avoid spoilers here but Vernor Vinge hits on pretty much exactly this in A Fire Upon the Deep. Though he's interested less in the hard sci-fi aspects of how/why and more on the consequences of it (story-wise).
tito 20 hours ago [-]
thanks!
kirubakaran 1 days ago [-]
For the opposite, "Dragon's Egg" by Robert L. Forward is a fantastic read
ggm 1 days ago [-]
I think this is a slow version of the quandry behind Quantum Computing: how do you distinguish events from the noise floor? It happens in the quantum context and it would happen in the millenial timeframe completing "operations" which have to be compared to e.g. the stability of orbit around a sun.
anigbrowl 24 hours ago [-]
'Large Language models? They can barely produce gibberish sentences, what would this tech ever be useful for?'
- bunch of people only ~4 years ago
Azantys 23 hours ago [-]
I questioned the speed not the output quality, that is another discussion
bibstha 23 hours ago [-]
It applies to speed too. The project paves way for more optimization at many layers overtime.
Francute 23 hours ago [-]
You can't improve what you can't measure.
Consider this like if it were the first test
winstonp 1 days ago [-]
16 tokens / s is not nothing.
throwaway219450 1 days ago [-]
It's the other way around due to poor framing, it would be much easier to compare if you [the repo] said 0.02 tps.
whatsThisBtn4 1 days ago [-]
Confusing numbers everywhere.
16tk/s... Then 3 tks per minute. Then someone else posted 0.3tk/s.
Azantys 1 days ago [-]
? Readme says 60-70s per token
nicechianti 1 days ago [-]
[dead]
acmnrs 1 days ago [-]
The title should probably be edited to specify "M1 Max" instead of "M1 Mac". You aren't running K3 on a base M1 anytime soon. Either way, still a very impressive project.
tito 1 days ago [-]
Done, Mac -> Max
mannyv 20 hours ago [-]
Going to try this on my M1 Ultra 128gb.
The point of these engineering tricks is to see the envelope of what's possible. You can use these tricks to both run a bigger model on smaller hardware or run a smaller model on smaller hardware.
nlessard 1 days ago [-]
Anyone who knows the state of NVMe hardware more than me know if this would obliterate the lifespan of your drive? Seems like the biggest limitation to me (some people are probably fine with letting their Macs churn over the weekend).
trollbridge 1 days ago [-]
No problem at all to read data over and over. In fact, LLM weights are a great candidate for low-quality flash that can't handle a lot of write cycles, and you want a large amount of storage cheaply...
addaon 1 days ago [-]
Reads are not generally life-limiting for flash. (Well, no more so than power-on time in general. You still have aging mechanisms like electromigration, but these are orders of magnitude slower than write-induced damage.)
nlessard 1 days ago [-]
Cool, thanks
mips_avatar 1 days ago [-]
Would be interesting to see how fast it would be on 4x mac studio 512gb machines.
ALLTaken 1 days ago [-]
Exactly my machine 64GB M1 Max
So happy about this! ♡
idk how people access (soldout) and even afford 512GB RAM MacStudio's. Isn't it $40k or so?
hmokiguess 1 days ago [-]
You're lucky, mine is the M1 Pro 16GB
teaearlgraycold 23 hours ago [-]
I just checked eBay. There's an insane price difference between used and new. $5k vs $40k.
ALLTaken 10 hours ago [-]
wow, that's crazy! But why? I know Apple will release a new model in September, but what do these guys selling jump onto?
I do really care to know: "Is there a new and more beefy solution we don't know about yet?"
knighthacker 22 hours ago [-]
Local AI is going to win. Not because it's cheaper btw.
jrhizor 24 hours ago [-]
Super cool, and I appreciate the upfront speed disclaimer
mindwok 24 hours ago [-]
Does anyone else feel like the writing is on the wall for a future of local models? Spamming data centres everywhere, powering them, having to commit insane capital to hardware, all the effort to serve inference over a network reliably - when here we are with a frontier model nearly running on a laptop.
Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at least, training is another story.
nomel 23 hours ago [-]
0.01 tk/s on an M1 Max is not "nearly". This is completely unusable, and in no way cost effective.
0.01 tokens per second means 1 million tokens ($3 worth of API usage [1]) takes 3.2 YEARS.
Ok in terms of running a 2.8T parameter model, that's true.
Looking more broadly though, a model I can run on my laptop (Gemma 4) is ~4 points away from GPT-5.3 codex or Sonnet 4.5 on arena.ai LLM leaderboard. Those models were SOTA less than a year ago.
hmokiguess 1 days ago [-]
Now set it up with an agent and a permanent `/goal` to say it cannot stop until it has solved for speed, then leave it on and livestream so we can all see when it becomes exponential. Could have the Eternal Jukebox playing in the background!
Anoian 1 days ago [-]
That sounds like the most boring exciting livestream of all time.
denysvitali 1 days ago [-]
60s/token - if only there was a way to drop that "s" this would be amazing
Says it requires a 2TB disk? Must it be internal NVMe?
Fergusonb 1 days ago [-]
You can use an external drive if it's mounted as a writable volume.
I would make sure it's fast, maybe thunderbolt 3/4/5 enclosure with a fast drive.
Xeoncross 1 days ago [-]
FYI, Thunderbolt 3 NVMe enclosures will be 10Gbps and 4 will be 40Gbps which is a big difference (and you'll notice it in the pricing as well)
tito 1 days ago [-]
The Github specifically mentions an option to stream it from an online host. It's extremely slow.
tjwebbnorfolk 1 days ago [-]
> ~60–76 s/token
I don't know if I'd call this "running"
sermah 1 days ago [-]
Had the same feeling when I first saw min/km units in some (human) running context.
UPD: I know it's not the same at all, just the reversal of units that gets me
tito 1 days ago [-]
I commented similarly below, but as a terrible programmer, I probably perform about 1 minute per token too (at Kimi 3 level). It puts into context how I think about intelligence
brokencode 1 days ago [-]
Maybe in terms of code produced, but one token is only a fragment of a thought for an LLM.
It’d be like thinking as slowly as Ents talk to each other in Lord of the Rings.
tito 1 days ago [-]
Oh, is that how it works? So, when somebody says a model is running at X tokens per second, it means that the thinking process is running at that, and output tokens are much lower then? Thanks to the explanation.
brokencode 24 hours ago [-]
It’s all just tokens to the model. Whether it’s thinking tokens or output tokens, they take the same amount of computation to produce. The only difference is whether the token is displayed to the user.
lostmsu 1 days ago [-]
under 0.02 tok/s
throwitaway222 22 hours ago [-]
how much money in hardware would it cost to get 100tps?
piterrro 1 days ago [-]
Will it fit on ESP32??
embedding-shape 1 days ago [-]
Better questions, how many ESP32s would it take to reach 1 tok/s decoding speed with K3?
tito 1 days ago [-]
1 button Kimi morse code interface
walrus01 24 hours ago [-]
tokens/second, no, more like seconds/token
ncallaway 23 hours ago [-]
The link beat you to it:
> It is not fast — about 16 seconds per token on our M1 Max
walrus01 23 hours ago [-]
At least it's not minutes/token?
onesandofgrain 1 days ago [-]
Cool gimmick
brcmthrowaway 1 days ago [-]
Why not train another smaller LLM to give the same answers as Kimi K3?
dkarras 23 hours ago [-]
why not zip the entire internet to 1MB so everyone can have a copy? because it is not possible - we don't know if it is possible. I mean we know it is impossible, but we don't know if it is possible to do it with acceptable quality loss.
Soon decent speed across two Mac Studios with 512GB of RAM.
And it gets worse with every token.
192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.
For AI agents this would take a year
This is like 98%.
If you already committed to your hardware, huge models at single or sub-digit TPS are still useful. LLM-as-judge is a good use case.
If your machine is gonna be idle overnight (and the power efficiency is excellent here), why pay openrouter if it’s not an interactive workload?
To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig.
Or could you do the same with custom layout of cheap slower RAM?
I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.
Optane would actually be useful in this era. Intel was ahead of their time.
Also its answering the question of what gonna happen if you wake up tomorrow and datacenters are gone. Or internets are gone.
Some people on our globe live in countries with no internet whatsoever. Of course most of them dont have Macbook with 64GB RAM either, but it's much much easier to get than internet connection or rack of GB200.
SOTA LLMs are efficiently compression of all the knowkedge humanity has built. Having ability to run it at home to extract said knowledge is important no matter the speed.
People tend to think they are the thriving hero but most won't survive the first year.
I'd much rather have a USB stick of prepper disk. It works on any shitty machine I find, and it's not going to take days for an answer on how to make potash or fix a solar panel.
But also that's a pretty extreme hypothetical. Imagine the polymarket on that.
Kimi Pen Pal. Bring back lettets and postcards. Do OCR, and use one of those 3D printer-like pen plotters write the model output as a letter.
Challenge would be automating the opening and OCR preparation, and the folding and mailing of the return letter. But given it's done commercially it should be possible.
Works in mutt; other MUAs may vary.
For example there's a billboard on 101 for Poppy Bank offering 4% interest and I asked my daily LLM to look into it through a voice note. The next morning, lo and behold it says it's an advertised rate, hard to actually get, and businesses aren't eligible. OK, done. Better than getting the response while I'm driving. And even if I wasn't driving, there's a level of, how do you say, it's easy to drop it the next day when I see it vs getting engrossed in the research.
It reminds me of when Willow Garage chose to name their bot the TurtleBot, because if they named it anything else, people would think it was fast and capable. But when they called it Turtle Bot, people just kind of liked it and were satisfied with what it did.
At the level of Kimi 3, I probably can code only about 1,000 good tokens per day, too. (thankfully coding isn't my job)
Need to justify buying an expensive rig that doesn't do what you expected.
Specifically thinking the people they could do something AI with cpu, and realizing it isn't feasible. Happened at my fortune 20 company. They had to get approvals and ofc it was useless. Plenty people tried to explain, but they were the principle engineer, and out ranked everyone.
"It's not going to work", the topic changed, and we never spoke about it again.
(note. I am not a believer in AGI)
"useful" is highly contextual. The clock of the long "now" is not useful in the sense you mean, to synchronise your wristwatch. I'm still glad it exists.
- bunch of people only ~4 years ago
Consider this like if it were the first test
16tk/s... Then 3 tks per minute. Then someone else posted 0.3tk/s.
The point of these engineering tricks is to see the envelope of what's possible. You can use these tricks to both run a bigger model on smaller hardware or run a smaller model on smaller hardware.
idk how people access (soldout) and even afford 512GB RAM MacStudio's. Isn't it $40k or so?
I do really care to know: "Is there a new and more beefy solution we don't know about yet?"
Local AI on your device seems like a much more likely future to me than datacenters in space. For inference at least, training is another story.
0.01 tokens per second means 1 million tokens ($3 worth of API usage [1]) takes 3.2 YEARS.
[1] https://www.kimi.com/resources/kimi-k3-pricing
Looking more broadly though, a model I can run on my laptop (Gemma 4) is ~4 points away from GPT-5.3 codex or Sonnet 4.5 on arena.ai LLM leaderboard. Those models were SOTA less than a year ago.
I don't know if I'd call this "running"
UPD: I know it's not the same at all, just the reversal of units that gets me
It’d be like thinking as slowly as Ents talk to each other in Lord of the Rings.
> It is not fast — about 16 seconds per token on our M1 Max