How do you feel about storing binary files with Git?

I think it's generally agreed upon that large files that change often do not belong while small files that never change are fine. But there's still a lot of middle ground where the answer is not so clear to me.

So what's your stance on this? Where do you draw the line?