DeepSeek's distilled new R1 AI model can run on a single GPU | TechCrunch
DeepSeek's distilled new R1 AI model can run on a single GPU | TechCrunch
DeepSeek's distilled new R1 AI model can run on a single GPU, putting it within reach of hobbyists.
So can a lot of other models.
"This load can be towed by a single vehicle"
7b trash model?
the Chinese AI lab also released a smaller, “distilled” version of its new R1, DeepSeek-R1-0528-Qwen3-8B, that DeepSeek claims beats comparably sized models on certain benchmarks
Most models come in 1B, 7-8B, 12-14B, and 27+B parameter variants. According to the docs, they benchmarked the 8B model using an NVIDIA H20 (96 GB VRAM) and got between 144-1198 tokens/sec. Most consumer GPUs probably aren’t going to be able to keep up with
Depends on the quantization.
7B is small enough to run it in FP8 or a Marlin quant with SGLang/VLLM/TensorRT, so you can probably get very close to the H20 on a 3090 or 4090 (or even a 3060) and you know a little Docker.
it's distilled so it's going to be smaller than any non distilled of the same quality
ew probably still censored.
The censorship only exists on the version they host, which is fair enough. If they're running it themselves in China, they can't just break the law.
If you run it yourself, the censorship isn't there.
Untrue, I downloaded the vanilla version and it's hardcoded in.
Yeah, i think the censoring in the LLM data itself would be pretty vulnerable to circumvention.
You can self host it right??
The self hosted model has hard coded censored content.
if the model is censored... then what, retraining it? Or doing it from scratch like what open-r1 is doing?