LLM for GTX1080 (8GB) for local use.
LLM for GTX1080 (8GB) for local use.
Could someone recommend a LLM for the Nvidia GTX1080? I've used the gptq_model-4bit-128g of Luna AI from the Bloke and i get a response every 30s-60s and only 4-5 prompts before it starts to repeat or hallucinate.
7
comments
try openorca-mistral-7b, it should fit in your GPU. Try using exllama2 to speed up interference.
thx, this one? https://huggingface.co/TheBloke/Mistral-7B-OpenOrca-GPTQ
7
comments