Restart an OOM killed docker automatically
Restart an OOM killed docker automatically
I got an home server that is running docker for all my self hosted apps. But sometimes I accidentally trigger Earlyoom by remotely starting expensive docker builds, which kill docker.
I don't have access to my server outside of my home network, so I can't manually restart docker in those situations.
What would be the best way to restart it automatically? I don't mind doing a full system restart if needed
Use
-mand limit the build job's memory so it doesn't kill the docker daemon.Fair enough. But I don't want a bandaid fix solution. Even more that I do all my docker through portainer and the option isn't there.
It could also be useful if a container got a memory leak and is unbounded
This isn't a band-aid, it's the literal fix.
Structuring the available CPU and Memory reservations for containers is LITERALLY the entire reason containers exist. Just because you're only familiar with the "dumb" way of using them doesn't mean you should be dismissive when someone offers you advice when you come here asking for it.
You're also seemingly just a dick for being lazy, because I looked, and wuddyaknow. So now you're just rude, dickish, and lazy.
Take the advice from the original responder, and then go and learn how to use the things you're asking for help with, along with some manners.
The other person may have responded with a fair amount of hostility, but they're absolutely correct. I run Kubernetes clusters hosting millions of containers across hundreds of thousands of VMs at my job, and OOMKills are just a fact of life. Apps will leak memory, and you're powerless to fix it unless you're willing to debug the app and fix the leak. It's better for the container to run out of memory and trigger a cgroup-scoped OOM kill. A system-wide OOM kill will murder the things you love, shit in your hat, and lick your face like David Tennant licked Krysten Ritter.
This is not a bandaid, this is the solution. What you try is, at least for this scenario, the band aid.
??? Your original proposed solution is literally a bandaid fix.
Systemd has config options for automatic restart of crashed services. https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#Restart=
Do you have your services set up with
restart=unless-stopped? I wonder if that would auto restart them after OOM.I don't know the best way but I would use cron and start docker every minute (if it's not running).
I don't know the best way
Apparently...
Don't do this. Either don't go OOM to begin with (somebody else told you how to limit container memory usage} and/or configure systemd to restart docker if it quits. I'm surprised systemd isn't already.
I'll try that. I know that systemctl has a start-or-reload command, but is there any "start-or-ignore" commands? Or start flags?
You should be able to make docker exempt from early oom. Check it's github for instructions.
But can it prevent killing only docker, and not the build/big containers processes?