I do worry about putting up public servers that other people might rely on because there's something I might not realize making it vulnerable.
So far I have pubkey root login only on the VPSs I'm messing around with, but my ol' reliable private key from 6 years ago might be beginning to fall behind on encryption standards.