42x Faster Prompt Lookup Drafting in llama.cpp
jadidbourbaki.github.io
Four changes to the n-gram caches of llama.cpp make drafting up to 41.6x faster, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.