heuristic is correct, but there is more to it. these models are MoE(mixture of experts) - essentially tinier models which are more specialised. at a given time, n out of m may be active (usually n is 2, i have heard there is not much increase with increase in n beyond 2). you can consider something like 1 expert is in ball park of 2B, and lets say there are 8-9 of them. then there are some extra “mandatory layers” which have to be there (you can assume that they orchestrate stuff)(at least that is how I have understood this). benefits of these “sparse” models is that at any given time, only n of them are being used, and hence compute is faster. most big models are moes (the deepseek 670+, or kimi 1000+), with size of experts never practically exceeding 30s (B). the largest open dense model afaik was the llama 405B, and below that some 100+ cohere and mistral stuff, but that was 1+ year ago (decades in ai terms). Industry found that moes were in general better bang for buck. moes have disadvantage that you have to hold all of the model in memory (not necessarily vram, ram is fine too, but ideally you should not be reading from disk) (the n active experts out of m keep on changing, so for each expert, if you have to read from storage, it is that much slower.), but actual compute happens with n experts, so it is fast. as a general rule of thumb, you can consider that a moe will always be stupider than its equal weight dense (single large model), but much faster. but training large model gets very hard. so you train many small models, and package together (you do not really train them separately, but that is a separate discussion).

now back to vram requirements - if using non quantised model, each param will take 1 or 2 Bytes of space in memory (fp8 or fp16). now 1 Billion bytes = 1 GB memory (not GiB), so 20B fp16 model would be ~40GB model. here fp16 is floating point number, with 16 bits. that is a representation of real number with finite precision. imagine like π extending to infinte digits, but usually you do not need all the digits (famously, nasa uses something like 15 or 16 digits after decimal) (in this case 16 bits include the mantisa and exponential and sign bit (1 for sign, maybe 11 bits for mantisa and 4 for exponential, but read about this online, it is not that important for current conversation)). but usually, you do not need all this precision while running the model. so we quantise these models (imagine rounding to lesser precision). a common quant is q4 (4 bits as opposed to 16). but there is some stuff we do not round that much, we do it to something like 8bits (these gives us the extra letters in q4"km"). Quantisation is not a free operation. what you gain in reduced memory usage, and slightly faster computation, you loose it in smartness, but trade off is usually worth it. You can measure the amount of smarts you loose by measuring something called “perplexity”, and yoy measure the amount of difference from the unquantised model. the lower the perplexity, the better the quant. usually, you do not want to go below q4. but for huge models (670/1000+), you have to use something like q1 or q2, because you got to do what you got to do to atleast run it. So you can imagine that q4km would roughly require 0.6 times number of params, so in this case, something like 12-13 GB. You have to hold all this in memory.

Ideally, both your context plus model weights should be held in vram, and size of context varies by model, but i usually do not think much of context (32k context requires close to 10GB of vram iirc, my stuff never usually goes beyond 4-6k).

many people can not afford much of vram, so run cpu and ram combos.

in this case, i have heard that with 8GB vram, and 64GB system memory, someone ran gpt oss 120b for roughly 10-11 tps, which is quiet fast imo, because active params ~6B.

but user here has a total of 40GB, so the best they can do is 32GB dense, so qwen3 32 B is good, but I think that would not be fast on that hardware, so i recommended something much smaller.

I prefer to preserve my anonymity, but that’s very kind of you ❤️

They don’t do IP detection, you account is saved in a cookie or with an email (I recommend you set an email so you don’t lose your account in case you clear your cookies)

Making two accounts is as easy at opening a private tab or a second browser to create another account. Just put two different emails and there you go, you’ll get 15% savings total. When your main account is empty, you can go on the second to use the 10% you earned back