There is nothing else open, but windows still won't let me delete the folder.

Process Explorer (the sysinternals tool) will show you which process has the handle open to the file.

I would say 'just reboot' but because I'm petty and also because I've had shit not release even after a reboot:

• Boot a live iso
• Mount the drive
• Delete that shit with the utmost prejudice
• Reboot
• Sunglasses (yeaaaaaaaaahhhhhhhh)

E: fancy formatting