Is it possible to run a reverse proxy only on a specific service or port?

That's the standard behavior. Read the documentation for whatever reverse proxy you want to use.

I just don't put the ports on the proxy config.

My method:

VPS with reverse proxy to my public facing services. This holds SSL certs, and communicates with home network through WireGuard link configured on my router.

Local computer with reverse proxy for all services. This also has SSL certs, and handles the same services as the VPS, so I can have local/LAN speeds. Additionally, it serves as a reverse proxy for all my private services, such as my router/switches/access point config pages, Jellyfin, etc.

No complaints, it mostly just works. I also have my router override DNS entries for my FQDN to resolve locally, so I use the same URL for accessing public services on my LAN.

First of all, doesn't Navidrome have authentication? So, I don't see why exposing it to the public is a problem.

Second, some reverse proxies support basic auth. This way, you can password-protect some services and is useful if the service does not have its own authentication. Here as an example snippet for Caddy:

example.com {
	basic_auth {
		# Username "Bob", password "hiccup"
		Bob $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
	}
	reverse_proxy myservice:8000
}

You'll have to look up the docs for other reverse proxies.

A typical use case is to forward a single port to the proxy, then set the proxy to map different subdomains to different machines/ports on your internal network. Anything not explicitly mapped by the reverse proxy isn’t visible externally.

A reverse proxy is basically a landing place that acts as a middle man between the client and the server. Most people set it up so that all traffic on 80 or 443 go to the reverse proxy, and then the reverse proxy gets the correct website based on the host header of the request.

If you are currently serving multiple websites on your server, then that means you are serving each website on a different port.

So, just make sure that the reverse proxy is serving on a port that is not used by your other sites. It will only respond on it's own port, and it will only serve the site(s) that you have configured in the proxy.

You'll be fine!

I’m afraid that when I install a reverse proxy, it’ll take my other stuff online and causes me various headaches that I’m not really in the headspace for at the moment.

If you don’t configure your other services in the reverse proxy then you have nothing to worry about. I don’t know of any proxy that auto discovers services and routes to them by default. (Traefik does something like this with Docker services, but they need Docker labels and to be on the same Docker network as Traefik, and you’re the one configuring both of those things.)

Are you running this on your local network? If so, then unless you forward a port to your server on the port your reverse proxy is serving from, it’ll only be accessible from the local network. This means you can either keep it that way (and VPN in to access it) or test it by connecting directly to your server on that port and confirm that it’s working as expected before forwarding the port.

How are you currently serving publicly?