I had an old house with one of those. I renovated the bathroom so I can confirm they all go into the wall.
God what a mess. 2ft of rusty used razor blades wedged in there.
Older medicine cabinets have a slot in them for this very purpose. A lot of people living in old homes probably have a razor blade slot or two and don't even realize it.