Hi all!

i have written a couple of posts in the past, i am an illiterate having fun with LLMs and AI in general, who is being pulled in in a deeper hole by the days…

I have extensive experience with Linux (Gentoo lover since 20 years here) i am a sw dev now “promoted” to management, and avid tech user, so not really illiterate, but i know very little about all this LLM game.

I started with OpenWebUI + Ollama and played as an idiot with random models. Then come across an NVIDIA RTX A4000 (16gb VDDR6) and plugged into my I7-8700 server with 64gb RAM. The server has a Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] too, unused at this time (server is 100% headless anyway).

I am currently installing LocalAI to run llama.cpp and improve my models capability and speed, planning to ditch OpenWebUI and Ollama, if LocalAI + llama.cpp works fine.

My first usage was chatting with random local models. Then i discovered Fooocus and quickly upgraded to ComfyUI. Last, i have set up my SubWave radio station and i am having so much fun…

I have a few questions:

  1. Can i leverage both my NVIDIA and the iGPU at the same time?
  2. If i use the iGPU do i need to fixedly allocate RAM from the BIOS to it? Or will it use system RAM as needed?
  3. Using llama.cpp i want to leverage also CPU usage, since i have 64gb ram (also shared by many more self hosted stuff, tough) is there anything special i need to do to achieve that?
  4. What are a set of models that you guys recommend for my setup? I am currently using qwen2.5-coder:14b-instruct-q5_K_M with ollama, and i am pretty satisfied with it’s coding capabilities, but i want something more general purpose for my SubWave (AI assisted web radio channel)
  5. I might have the opportunity to install a second RTX A4000, identical to the first, on my server (need to check pci-e slot availability and power supply specs), would that make any sense at all?
  6. Power consumption wise, do the NVIDIA cards suck power also when not in active use?
  • brucethemoose@lemmy.worldEnglish
    5·
    4 months ago

    The best you could run on that system, I believe, is Step 3.7 via ik-llama.cpp.

    You don’t want, nor need, the iGPU. Unfortunately nothing is really optimized to use it for hybrid inference, and it’s not very powerful anyway. You’re better off with a CUDA/CPU backend.

    How it would work in practice is:

    • The “dense” parts of a MoE model like Step stay on the GPU.

    • The big, sparse parts stay in CPU RAM.

    • For prompt processing, everything is done on the GPU since the penalty for going over the PCIe bus isn’t too bad.

    • For text generation (generating the actual output), your CPU cores run the sparse part. This is fine. It’s not nearly as slow as you’d think, as this part is really compute lite and AVX accelerated.

    The tricky part is finding appropriate models and quantizations for your setup. You’re looking at something in the 100-170B range, basically, with the experts quantized to 3-4 bits and the dense weights at 6-bit or higher.

    I can look around later, if you’re interested, but I haven’t kept up with model releases in this range other than the Stepfun series.

    On a separate note, I wouldn’t containerize ik_llama.cpp.

    It just wastes precious RAM and compute. It has basically no dependencies, so it’s not really a security risk, and there isn’t much good that bundles models with it.

    Now, the UIs. Containerize those. Those have tons of dependencies, so it’s easier and safer.

    • Shimitar@downonthestreet.euOPEnglish
      1·
      4 months ago

      Not clear to me where to find Qwen 3.6 35B-A3B, at Q4_K_M, and if that will leave free vram for smaller models (tts)

      I am running LocalAI that manages for me llama.cpp plus at least another backend for the tts and I cannot find that model on the list, but it can import models downloaded manually

    • Shimitar@downonthestreet.euOPEnglish
      1·
      4 months ago

      Understood, so my iGPU is too old, and i am better off with my NVIDIA only. Also, putting two might be just a waste, and i doubt my PSU can handle them even if i had the proper PCIe slot.

      So, using llama.cpp if i use a model bigger than my VRAM will offload to CPU? I am not fully clear on this

        • Shimitar@downonthestreet.euOPEnglish
          1·
          4 months ago

          I am trying to understand if it make sense to run two cards.

          I am using ai for chat and to support my subwave radio station (ai dj + music tagging and text to speech, small models are fine) so I am going to test a qwen3.5 9b derived model to have free ram to run text2speech model as well.

          The second usage for ai is image generation, I am using ComfyUI so it’s not clear to me how that plays with the other models… Will they be unloaded when I use comfy? Or will comfy just fail or offload to CPU?

          So, if having two cards pinned one to LocalAI and one to ComfyUI is a good choice, I can look into the hardware doubts. If nothing would practically change, no …

            • Shimitar@downonthestreet.euOPEnglish
              1·
              4 months ago

              Yes. All you say make sense. Tts can and shall go on CPU, who cares.

              It’s handling both general chat and ComfyUI the biggest point, but to be honest o don’t need them both at the same time. The point is how easy is to switch since this is an unattended server, shutting down containers might not be the easiest approach. Specially if some other family member want to use it.

              I will switch screen out on the igpu, it’s unused anyway, and see if I can run one card only, but I am struggling in running almost any model on LocalAI with llama.cpp I need to dig in more time on the issue.

              Adding the second Nvidia will require a new psu, the mobo has the slot, even if it’s only 16x downgraded to 4x according to Claude it will work just fine for normal inference. But I will keep it as a last resort, I don’t particularly enjoy fiddling with that bios.

              In any case I want both the llm and comfy to run on GPU and not offload so CPU.

  • Eggymatrix@sh.itjust.worksEnglish
    31·
    4 months ago

    Qwen3.6-27b and qwen3.6-35b-a3b are your starting point. Ask those how to configure your stuff. You can then move to something larger like a 120b model using some offloading if you want.

    Run pi.dev and llama.cpp, use always the latest builds or compile from source.

    The card does not eat energy when it is not in active use.

  • brucethemoose@lemmy.worldEnglish
    1·
    4 months ago

    Actually one addendum/correction:

    You can use your integrated graphics to render the desktop. Plug your monitor into the UHD graphics instead of your Nvidia card.

    This saves a notable about of VRAM you can use to fit more context, and speeds up inference a bit too. It also lets you configure models closer to your VRAM limit, as you no longer have the variability of apps randomly taking it.

    • Shimitar@downonthestreet.euOPEnglish
      2·
      4 months ago

      Yes,but that is a headless server anyway. So I will indeed switch to using the igpu as primary video sooner or later (need to physically go to the server, I have a network Kem but that would be pretty useless after I get into bios and change the video output)