A note that this setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you’d need a dual socket EPYC server motherboard with 768GB of RAM.
A note that this setup runs a 671B model in Q4 quantization at 3-4 TPS, running a Q8 would need something beefier. To run a 671B model in the original Q8 at 6-8 TPS you’d need a dual socket EPYC server motherboard with 768GB of RAM.
Well if you’re downloading, copying or creating large models that are several hundred GBs you’re going to want a normal drive. SMRs have a small staging area and once that is full it has to start re-ordering the data on the platters. Once your drive is in the process of re-ordering your write speeds are going to make it look like a failing floppy disk. I had a large file copy operation (>1TB) to a RAID pool of SMRs take like 16 hours. And I also found out that my backup drive is SMR because it took several days to do a full backup from scratch, which caused me to look up its detailed specs.
It always starts out looking great but eventually the staging area will get full and then your CPU will spend most of its time twiddling its thumbs until a chunk of staging area becomes available again; Repeat until operation is done. The greatness of Shitty Magnetic Recording.
If you want to know what recording method your drive uses, you grab the model number from:
# smartctl -a /dev/YOURDRIVE | grep "Device Model"and look that up in a search engine. It should either lead you to a Data sheet or the manufacturers website where they list the specifications.
If durable and large SSDs were more affordable where I am I’d slowly replace all my spinning rust. But right now HDDs are overall still the better option for me, at least for mass storage.