I'm generalising here and obviously it won't apply to everyone, but I tend to think most people with small phones look at huge phones and say "whoa, too big" and thus never bother to give it a go. People with large phones either love them already, or they love the idea of a small phone — so they go get one, and in most cases, immediately miss their large phone.
Of course, some people have experienced both formats enough to know that they definitely prefer small phones — but, unfortunately, they appear to be a minority. Small phones don't sell well, or else there'd be more of them. And so, these people fall into that all-too-common market category 'You're Not Significant Enough'.
All that said, it absolutely depends on what you want from your phone. I use my S23 Ultra to edit photos and videos, play a few games, get some work done — and all of those things benefit from a big display. Hell, even typing is easier for me on a big screen, thanks to my sausage thumbs. 😂
Of course, some people have experienced both formats enough to know that they definitely prefer small phones — but, unfortunately, they appear to be a minority. Small phones don’t sell well, or else there’d be more of them.
The conundrum with sales though is, if almost no major brands are selling them, how would a regular person ever find them? In the Android market, there are few companies making & selling them & they arguably aren't widely popular for phones (sorry Asus, no surprises Unihertz) , so almost inevitably they won't sell well.
In the iOS market, they still have the iPhone SE which I think is the smallest of that lineup, and I don't think Apple would continue to make those if there wasn't still a market for them. Also to swing back to the first part of what I've quoted here, I guess I'm in that minority.
I recently got a new phone as I was concerned about the battery of my old one, but it turned out I may have gotten ahead of myself, so I have a very clear point of comparison between the two when it comes to their form factors and uh...Yeah, when you do it's no contest. The smaller device is way more comfortable to use, as I can reach most of the screen with one hand without feeling like I'm about to fumble it & drop it.