[HELP] In GPT4All settings, selecting AMD graphics card yields no performance improvement over CPU
[HELP] In GPT4All settings, selecting AMD graphics card yields no performance improvement over CPU
Background: This Nomic blog article from September 2023 promises better performance in GPT4All for AMD graphics card owners.
Run LLMs on Any GPU: GPT4All Universal GPU Support
Likewise on GPT4All's GitHub page.
September 18th, 2023: Nomic Vulkan launches supporting local LLM inference on NVIDIA and AMD GPUs.
Problem: In GPT4All, under Settings > Application Settings > Device, I've selected my AMD graphics card, but I'm seeing no improvement over CPU performance. In both cases (AMD graphics card or CPU), it crawls along at about 4-5 tokens per second. The interaction in the screenshot below took 174 seconds to generate the response.
Question: Do I have to use a specific model to benefit from this advancement? Do I need to install a different AMD driver? What steps can I take to troubleshoot this?
Sorry if this is an obvious question. Sometimes I feel like the answer is right in front of me, but I'm unsure of which key words from the documentation should jump out at me.
My system info:
- GPU: Radeon RX 6750 XT
- CPU: Ryzen 7 5800X3D processor
- RAM: 32 GB @ 3200 MHz
- OS: Linux Bazzite
- I've installed GPT4All as a flatpak
I am somewhat new to Linux and hosting local LLMs, but I think I had to install AMD ROCm for LLMs to work with my GPU.
https://rocm.docs.amd.com/en/latest/about/release-notes.html
Can gpt4all use ROCm?
Rats—according to their System Requirements (Linux) page, they don't support Fedora. Even if I were to switch to a supported distro, it looks like only a small set of graphics cards are supported, and unfortunately, mine is not one of them. 😢
Supported graphics cards:
Supported operating systems:
Thanks anyway for the tip!
My best guess would be that you installed the flatpak version of gpt4all, and somehow that messed with it's ability to use the GPU.
Vulkan should work with your GPU, the model you chose should fit in your GPU if it is q4_0 or q4_1 which is the default in gpt4all I think.