YSK: Google keeps a detailed history of your location. You should delete it.
Why YSK: So you can turn off location tracking in Google
Go to the My Activity settings in your Google account and turn off "Location History". The Google and Google Maps apps record your location history by default.
Contrary to this, this saved my life once as I was able to prove I was not present for a meeting many at my office got nicked for.
Rather than erase your data, seek to control it. I'm a former data protection officer for a data broker, you can never really hide without disconnecting; rather it seems better to control the data you put out.
Edit: without sounding like an obtuse nonce I mean to say it's better to assume they have this data no mater what you select, if you want to limit their access spoof your location and feed them garbage data
I have a very similar story. I was able to avoid a fine because I was able to prove I was not in a particular location at a particular time simply by calling up my "timeline", which is what they call your location history in G Maps.
I agree that there is always going to be a trade-off between privacy and features. You should absolutely know the trade-offs you are making, and decide if they are worth it for you.