Lemmy: Bestiverse
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
RSS BotMB to Hacker NewsEnglish · 11 days ago

A 35B MoE on a 16 GB GPU, without the offload tax

www.lucebox.com

external-link
message-square
0
link
fedilink
3
external-link

A 35B MoE on a 16 GB GPU, without the offload tax

www.lucebox.com

RSS BotMB to Hacker NewsEnglish · 11 days ago
message-square
0
link
fedilink
Luce Spark: a 35B MoE on a 16 GB GPU, without the offload tax
www.lucebox.com
external-link
A 33-35B MoE only fires ~8 of 256 experts per token, but holding it on the GPU costs you all of them. Luce Spark pins the experts your traffic uses, offloads the rest to CPU, and decodes the whole token in one fused graph, so a 33-35B MoE fits a 16 GB card and still runs near full-GPU speed (~100 tok/s vs ~119 all-GPU on a 3090). Self-tuning, one flag.

Comments

alert-triangle
You must log in or # to comment.

Hacker News

hackernews

Subscribe from Remote Instance

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !hackernews@lemmy.bestiver.se
lock
Community locked: only moderators can create posts. You can still comment on posts.

Posts from the RSS Feed of HackerNews.

The feed sometimes contains ads and posts that have been removed by the mod team at HN.

Source of the RSS Bot

Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 685 users / day
  • 1.81K users / week
  • 4.04K users / month
  • 9.92K users / 6 months
  • 2 local subscribers
  • 5.04K subscribers
  • 54K Posts
  • 29.2K Comments
  • Modlog
  • mods:
  • patrick
  • RSS Bot
  • BE: 0.19.15
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org