Self hosting LLMs on a remote VPS
Self hosting LLMs on a remote VPS
Hi all, I'd like to hear some suggestions on self hosting LLMs on a remote server, and accessing said LLM via a client app or a convenient website. Either hear about your setups or products you got good impression on.
I've hosted Ollama before but I don't think it's intented for remote use. On the other hand I'm not really an expert and maybe there's other things to do like add-ons.
Thanks in advance!
Do you have lots of money? Cuz that's going to cost lots of money. Just get a cheap GPU and run it locally.
That depends on the use-case. An hour of RTX 4090 compute is about $0.69 while the graphics card is like $1,600.00 plus computer plus electricity bill. I'd say you need to use it like 4000h+ to break even. I'm not doing that much gaming and AI stuff, so I'm better off renting some cloud GPU by the hour. Of course you can optimize that, buy an AMD card, use smaller AI models and pay for less VRAM. But there is a break even point for all of them which you need to pass.
No, but I have free instance on Oracle Cloud and that's where I'll run it. If it's too slow or no good I'll stop using it but there's no harm trying.
They also offer some templates, instructions and blog posts about this. And since I'm not advertising for a single company, there's also vast.ai and several others.
thanks. I'll give it a try
Ollama and openwebui for a nice web interface.
this looks very good I'll try this at home
Me personally, i use my AMD 7700XT to run ollama on my main pc. Can be helpful for troubleshooting as the internet gets worse amd worse to search especially if i dont know what the issue is. Thats my main use case for it but id like to set up something with RAG and use it to help me with documentation if i have questions.
I don't think its super worth it to use a VPS for an LLM if you already have a decrnt gpu that you can run it on. If yoy dont already have the hardware, plenty of older gpus can run the models pretty well. My 1070ti still kicks ass all these years later and you can find them for $100 bucks or less on ebay used. Ive used it for ollama as well and it does just fine.
Will it be super fast or a really big model? No, but if its for personal use, i dont see any benefit to paying a monthly subscription for it and like i said, works well enough for me.
Its also more secure and private to host it yourself if thats worth anything to you. Thats one of the biggest reasons i self host.
If you do set up a RAG store, please post the tech stack you use as I’m in a similar situation. The inbuilt document store management in ollama+openwebui is a bit clunky.
If i can figure it out I'll be sure to post something lol.
So far, i found a python project that is supposed to enable RAG but i have yet to try it and after a reinstall of my linux pc to Popos, im having less than success getting ollama to run.