• brucethemoose@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    9 days ago

    Jamba is a killer model flying under the radar, though it does have a quirk I more recently discovered: no prompt caching in llama.cpp (yet).

    If you have a 24GB GPU you can cram Nemotron 49B in it with no offloading, including the new reasoning version. It’s a monster at STEM stuff, and I can upload my special quantization (3bpw, with 4bpw KV heads, exllamav3) if you ask.

    Qwen 30B coder is ridiculously fast for how smart it is at coding, just came out today…

    TBH the last week or two has been nuts with new releases.

    But FYI if you are looking for pure prose quality, I still use EVA Gutenberg 32B (based on Qwen 2.5 base) and Jonboro’s brand new QWQ 32B fine tune, as new models have not surpassed them IMO. But for creative writing, I tend to write novel style instead of multi turn, so YMMV.

    • swelter_spark@reddthat.com
      link
      fedilink
      English
      arrow-up
      2
      ·
      7 days ago

      I only have a 16GB card, and my CPU is new enough that it’s better to offload some layers of all but 7-8B models, so I haven’t tried exllama, but you’re making me think I should, if only for comparison.

      I like Qwen 2.5 based models in the 14B size range, but I don’t think I tried the bigger ones. I tried the QWQ and didn’t really like it, but I haven’t seen this new one. You’ve given me a whole list of things to try, so thanks.

      • brucethemoose@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        7 days ago

        16GB

        Is it 3000 series or newer?

        If so, with exllamav3, you can squeeze 32Bs in that 16GB card with relatively little loss. For instance: https://huggingface.co/turboderp/EXAONE-4.0-32B-exl3/tree/3.0bpw

        The 3bpw weights are 13 GB, say another 1.5GB for some q5_q4 context, and you are looking at 14.5GB-15GB or so. It will be tight, but it will be leagues smarter than 14Bs.

        24B Mistral models will fit much more easily. No need to CPU offload those on a 16GB card, you just need to be careful with your settings.

        • swelter_spark@reddthat.com
          link
          fedilink
          English
          arrow-up
          1
          ·
          13 hours ago

          I have an rx 6800. Looks like exllamav3 doesn’t support AMD cards yet… I’ll keep an eye on it, so I can try it when ROCm or Vulkan support is added.

          • brucethemoose@lemmy.world
            link
            fedilink
            English
            arrow-up
            1
            ·
            edit-2
            12 hours ago

            Ah. You can still run them in exllamav2, but you’re probably better off with ik_llama.cpp then:

            https://github.com/ikawrakow/ik_llama.cpp

            It supports special “KT” quantizations, aka trellis quants similar to exllamav3, and will work with vulkan (or rocm?) on your 6800.

            Quantizing yourself is not too bad, but if you want, just ping me, and I can make some 16GB KT quants, or point you to how to do it yourself.

            It’s also a good candidate for Qwen3 30B with a little CPU offloading. ik_llama.cpp is specifically optimized for MoE offloading.