GitHub - chiennv2000/orthrus: Fast, lossless LLM inference via dual-view diffusion decoding.
github.com
Fast, lossless LLM inference via dual-view diffusion decoding. - chiennv2000/orthrus
  • BB84@mander.xyzOPEnglish
    3·
    3 months ago

    My oversimplified and possibly wrong understanding: this is like speculative decoding, but instead of a separate draft model (which does its own prompt processing), they use some diffusion thing strapped on top of the main model. The diffusion reuses the high-quality prompt processing result of the main model.

    The 7.8x faster claim sounds almost too good to be true. But even if we get like 3x then this is still a huge revolution in localLLMing.