Width vs. depth: speculating on the margin | Doubleword
blog.doubleword.ai
Some thinking about how to trade off batching and speculative decoding in a running inference engine.