8700g with 256gb ram is possible on desktop. Half the APU performance, but less stupid bigger model > fast, for coding. No one seems to be using such a rig though.
To be fair: Motherboard, cpu and ram I bought 6 years ago with an nvidia 1660. Then I bought the Radeon RX 6600 XT on release in 2021, so 4 years ago. But it’s a generic gaming rig.
I would be surprised if 2000$ worth of modern hardware, picked for this specific task would be worse than that mini PC.
Seems pretty decent, but I wonder how it compares to an AI optimized desktop build with the same budget of 2000$.
It will probably kick the ass of that desktop. $2000 won’t get you far with a conventional build.
8700g with 256gb ram is possible on desktop. Half the APU performance, but less stupid bigger model > fast, for coding. No one seems to be using such a rig though.
Well, thats what I said “AI optimized”.
Even my 5 year old 900$ rig can output like 4 tps.
With what model? GPT oss or something else?
LLama 3 8B Instruct: 25tps
DeepSeek R1 distill qwen 14b: 3.2tps
To be fair: Motherboard, cpu and ram I bought 6 years ago with an nvidia 1660. Then I bought the Radeon RX 6600 XT on release in 2021, so 4 years ago. But it’s a generic gaming rig.
I would be surprised if 2000$ worth of modern hardware, picked for this specific task would be worse than that mini PC.
To be honest that is pretty good. Thanks!
I promise. It’s not possible. But things change quickly of course.
(Unless you’re lucky/pro and get your hands on some super cheap used high end hardware…)
There is nothing “optimized” that will get you better inference performance of medium/large models at $2000.