No, I lived in Japan and have visited all over. It's more an issue in men's bathrooms than in women's to not have soap, based on my conversations. In women's bathrooms, it's common to not have toilet paper, and people carry around tissues. This is more of a city thing than a rural thing. In the cities, people pass out tissue packets with advertisements in them (as a job) and people carry those around and use them in the restrooms.
Edit: It might be the case that places which have more tourists don't have this problem as much. That's still not my experience, however.
As I recall when I lived near Shizuoka, Osaka had a reputation for going their own way. The main difference I remember is getting off the train and everyone walked to the other side than typically done elsewhere. I would have liked more time in Osaka, it seemed more vibrant and open in some ways.
I was just in Japan for about 4 months, mostly Tokyo. Id say somewhere around a quarter of public men's rooms I used didn't have soap dispensers. Taiwan was worse though - most baffling was the lack of soap on my plane to and from Taipei