Ansible Playbook - How do I reverse engineer a running system?
I have a few VMs and PMs around the house that I'd setup over time and I'd now like to rebuild some, not to mention just simplify the whole lot.
How the hell do I get from a working system to an equivalent ansible playbook without many (MANY) iterations of trial & error - and potentially destroying the running system??
Ducking around didn't really show much so I'm either missing a concept / keyword, or, no-one does this.
I would copy the existing system onto a new system:
Update system to the latest packages
Create a new base system using the same distro
Check which packages are not on the new system, add them to your playbook
Install packages on new system
This will take some time. Run a find of all files and pass them to md5sum or sha512sum to get a list of files with their checksum. Compare the list from the old system to the new system.
Update your playbook with these findings. Template is probably the way to go, Lineinfile might be good as well, use copy if nothimg else works.
Check firewall settings and update your playbook.
Anyhow this will take some iterations, but while you have a copy of your ‘production’ system, you can test on your ‘test’ machine until you have the same functionality.
What I did to get rid of my mess, was to containerize service after service using podman. I mount all volumes in a unified location and define all containers as quadlets (systemd services). My backup therefore consists of the base directory where all my container volumes live in subdirectories and the directory with the systemd units for the quadlets.
That way I was able to slowly unify my setup without risking to break all at once. Plus, I can easily replicate it on any server that has podman.
No, I keep that private to minimize the information I leak about what I host, sorry. (I also don't do git-ops for my server; I back the mentioned directories up via kopia so in case of recovery I just restore the last working state of data+config. I don't have much need to version the configs.)
the first step is workout what you did, what did you install and where from. Then what config files got edited.
Much like a playbook for a disaster recovery test
Next is using some of the builtin modules like package and copy, make a very noddy playbook that will install stuff and copy up the config. if you have vms, you can use a test vm to see if the playbook works.
You will need many iterations of trial and error. No way.
You can speed up testing your playbook by using Molecule or something similar. Don't touch your working VMs until you get a service (role) set up correctly in your test environment. If you need to set up multiple services in a single VM, you can automate their deployment sequentially, of course.
P. S. I don't like Ansible and won't recommend it because it is full of bugs and non-obvious behavior. However I didn't investigate alternatives and can't suggest a better one.
Could you elaborate a little bit about "full of bugs" and "non-obvious behaviour"?
I use Ansible at work for a couple of years already and never encountered anything like that. (I have about 10 playbooks, about 30 roles, about 20 linux servers that I administer)
Same question. But with 100s of playbooks, and thousands of servers. This feels like someone had a bad experience with their first 30 minutes of ansible and gave up before looking at the command reference.
No, I can't. I use it only occasionally, so I don't remember everything. But many times configurations didn't work as described in documentation and I had to find a different way to achieve a required result. Sometimes this behavior changed from release to release. This thing doesn't seem something that I can rely on. But we use it in our company many years, so switch to another tool would be painful.
afaik, there's no way "convert" a running system into a playbook.
I'd recommend looking at what your systems have in common (installed packages, timezone etc...) and create playbooks based on that and work your way up from there.
@Cyber Yeah it's gonna be pretty manual as others have mentioned. Some areas to look at:
- Filesystem provisioning, mounts, etc.
- Packages
- Users, groups
- Time zone, locale language, time format etc.
- /etc/
- /root/ and /home/
- SSH settings
- Services
- Cron jobs/systemd timers
There is a bit of overlap between some of those categories. Some bits are going to see more or less use on VMs vs physical. And remember that in ansible there are built in modules for a lot of functionality.
@Cyber If you have some old wiki notes on how the system was setup originallythen it night be easier to ignore the current system and translate the wiki instructions into ansible. Still manual, but easier than reverse engineering. Another thing you can look at is bash history. Apart from backing up/cloning the system before you start I would also get a copy of the bash history for the various users and add it to a wiki or issue too. It will be useful.
@Cyber Yeah the bash defaults are incredibly limited by default, something like 1000 entries, 2000 bytes. I always make those something like 100,000 and 1MB. So the defaults can definitely bite you on an existing system, it may not have stored every command. https://superuser.com/a/664061