I do this too. I do not like anything on my Desktop, but I download files to there, which forces me to deal with them. Interestingly enough, my Downloads directory is a barren wasteland.
Desktop is my temp space. There are files with a very limited shelf live (logs I downloaded to search something, screenshots, ...) so I have to clean them up on a regular base before my desktop becomes too crowded and I get annoyed.
That requires effort up-front every time I click download.
I'd rather just click 'download' and let it go to an easy to find default location. The desktop means I won't just forget about it for months. It may sit there for a day or two, but it definitely won't get ignored the way a folder I rarely look at does; because the clutter right in my face annoys me into cleaning it up.