I once set an S3 lifecycle setting that accidentally affected 3 years worth of logs to Glacier. The next morning I woke up to a billing alert and an AWS bill with an extra $250k in charges (our normal run rate was $30k/month at the time). Basically I spent my entire add annual cloud budget for the year overnight.
Thankfully after an email to our account rep and a bunch of back and forth I was able to get the charges reduced to $4,300.
Deleting from. We move logstores and I added an ageout policy for anything over 1 day, to "easily" empty a bucket overnight. I forgot that I had been cycling stuff to glacier after 6 months, and there were 3 years of logs in there.