• 0 Posts
  • 121 Comments
Joined 3 months ago
Cake day: March 24th, 2026
  • melfie@lemmy.ziptoLocalLLaMA@sh.itjust.works•Qwen 3.6 27B running at 46 tok/s on an RX 9070 XT (llama.cpp + MTP Speculative Decoding is basically magic)English
    1·
    8 days ago

    I’m running the TurboQuant fork of Llama.cpp and doing K:Q4, V:Q3 with 200k context on the Qwen 3.6 MoE variant. On a RTX 3070 with CPU offloading, I’m getting 280 t/s prefill on over 100k context and about 20-30 t/s decode. It’s usable, though the MoE makes a lot of coding mistakes and I have to make it fix things constantly.

    I just tried the 27B variant yesterday and it was too slow to be usable with 50 t/s prefill and 0.27 t/s decode, though it fixed a bug the MoE model was struggling with in one shot (took hours 😭). I bet with 16GB of VRAM and TurboQuant for KV cache, you’d get decent speeds and decent context. I have a RTX 4060 in my server and I’m considering moving it so the 3070 and 4060 are in the same machine to see how 27B does across 2 cards with 16GB.

  • melfie@lemmy.ziptoTechnology@lemmy.world•A $200 ChatGPT subscription could cost OpenAI $14,000 if you actually used it to its full potentialEnglish
    1·
    10 days ago

    I’m running Qwen on my own hardware.

    I haven’t found anything yet that isn’t in its training data that I’d want it to evaluate as a control group, but you’re right that it would be a useful exercise.

    Here are some examples of the feedback it has given me:

    • This plot point hasn’t been “earned” and needs more setup to pay off properly

    • This dialog is an exposition dump. Find a better way to show, not tell.

    • This character feels like a vehicle for jokes, and isn’t developed enough.

    Most of the advice I’ve gotten so far relates straight back to what I’ve read in writing books and is pretty cut and dry. Some things are a matter of opinion, and I push back when I disagree or when I am deliberately breaking a rule.

    Edit:

    To your other point, you’re correct that a LLM saying something is good doesn’t mean humans will think so, or vice-versa. A LLM is but one tool in the process, and doesn’t replace real human feedback. For example, with a comedy, do human readers laugh out loud when reading it? A LLM can determine statistically whether something is intended to be a joke and whether the joke is overused, etc., but can’t tell you if the joke is actually funny.

  • melfie@lemmy.ziptoTechnology@lemmy.world•A $200 ChatGPT subscription could cost OpenAI $14,000 if you actually used it to its full potentialEnglish
    2·
    10 days ago

    I have read a lot of books and do analyze my work in terms of techniques and principles I’ve studied over the years. However, even top professional writers don’t work in a vacuum. TV writers, for example, have “the room” with a team of professional writers, producers, etc. weighing in on all writing decisions. For indies, you don’t have that luxury, and even getting another human who is good at writing to read what you wrote and share detailed feedback is hard, especially when said humans aren’t getting paid to do it full time. Asking friends and family to critique your writing will often result in them trying to spare your feelings, whereas Qwen will happily rip your work to shreds and not care if it just shit all over your passion project.

  • melfie@lemmy.ziptoTechnology@lemmy.world•A $200 ChatGPT subscription could cost OpenAI $14,000 if you actually used it to its full potentialEnglish
    3·
    10 days ago

    It’s not my “coach” any more than random people online would be if I posted it in a forum somewhere and no more than a LLM or a human peer reviewing my code is my “coach”. It provides a different perspective to help me see beyond my own biases with feedback I can accept or reject.

    Qwen has obviously been trained on writing books and a ton of screenplays. As an experiment, I changed the character names in a classic sitcom script and it was able to identify the series from the writing style and then it also identified the episode. It’s not useful for doing the actual writing, but it does provide useful feedback based on sophisticated statistical analysis of my work compared to its professionally-written training data.

  • melfie@lemmy.ziptoTechnology@lemmy.world•A $200 ChatGPT subscription could cost OpenAI $14,000 if you actually used it to its full potentialEnglish
    22·
    10 days ago

    I have code for personal projects that solves problems in novel ways as well as other creative work that I don’t care to let Anthropic and OpenAI train their models on. Is my work worth $14k to them? Well, the value is intangible to me, and I can say at least that companies have paid me a lot more than that for code that took a similar amount of time to write. The major data sources for training LLMs like GitHub, Reddit, Wikipedia, etc. have already been tapped, but they always need more and more data. If you want to give them your data like it’s not worth anything, you do you, but they’re not getting mine. If I need LLMs for personal use, it’s local or nothing.

  • melfie@lemmy.ziptoTechnology@lemmy.world•A $200 ChatGPT subscription could cost OpenAI $14,000 if you actually used it to its full potentialEnglish
    5·
    10 days ago

    I started out using GitHub Copilot at work because there was a lot of pressure to use AI, and I was put off by how we were churning through PRs that seemed to work, but having to go back and fix the slop afterwards.

    Now I’ve realized that there are skillful ways and unskillful ways to use LLMs, and they can in fact be a useful tool beyond just generating slop. They don’t replace a human thinking critically, but they can automate mundane, routine tasks. They can also summarize text well and suggest options for humans to consider. For example, LLMs reviewing code will often find issues the human reviewers missed.

    In addition to coding, I’ve recently been using Qwen locally for screenwriting. It can’t write worth a shit, but it does a good job critiquing my work and pointing out problems with the story structure and the like. For example, I can tell it something like “look at the 7 plot elements described in this MD file and point out where this story does and doesn’t follow this structure”, and the output is quite useful.

    While LLMs aren’t the magical silver bullet the tech bros are hyping them up to be, they can still be a useful tool. If they’re just used to generate slop, then no, they’re worse than useless.