If in the '80s we fought to squeeze a program into a few kilobytes, we now have to squeeze the most useful LLM work into the tokens we can afford. You ready?
Not doing stupid shit like asking an LLM to scan a million line file when you could grep, breaking skills files into progressively-discoverable chunks, running normal algorithms where a full LLM isn’t necessary.
And then using an open source model to really reduce the cost
Depends on how things play out. A combination of prices, regulations, and public sentiment leads a lot of petro-companies to offer energy-saving advice (even if it hardly offsets the rest of the negatives of the business, they do wind up doing this)
10x token usage reduction is often achievable.
Not doing stupid shit like asking an LLM to scan a million line file when you could grep, breaking skills files into progressively-discoverable chunks, running normal algorithms where a full LLM isn’t necessary.
And then using an open source model to really reduce the cost
Yeah, but none of that stuff enriches AI companies, so it won’t be recommended by them
Depends on how things play out. A combination of prices, regulations, and public sentiment leads a lot of petro-companies to offer energy-saving advice (even if it hardly offsets the rest of the negatives of the business, they do wind up doing this)