It took a lot of work to get PC parts to become interoperable. There's all kinds of special negotiations that happen at boot to discover the state of hardware. There are standard drivers for most peripherals
Mobile devices aren't expected to change hardware so everything is hard-coded into the ROM. No discovery protocols are supported. Standardized drivers make it hard to ship new features or squeeze every last bit of performance so no effort goes into making them
Also, a lot of people are not aware that the PC world was even more wild west before this standardization than the current state of smartphone.
I do hope this standardization would come soon. You'd with all this collection knowledge the humanity has, people would come up with a more uniform solution...
The problem here isn’t knowledge, but incentives. Like, someone could design and start manufacturing a phone with very standard stuff, but it wouldn’t sell except for a dozen enthusiasts. Even on PC, Linux isn’t as widely used as Windows on consumer hardware, as there’s no focus on user experience. For a phone like that to work, they would need to solve a problem most other phones don’t solve.
PC standardization took a few decades but there were not many adopters initially (compared to the total population) - mobile is different, basically everyone everywhere has a mobile device, so standardization will likely be much faster
Because the nuance of calling a PC a PC is that it supports a standard motherboard configuration and any Linux kernel will be able to boot. Mobile devices aren't like this and every component requires a specialized driver to work.
And just to add to that, anyone who has installed Linux before knows that everything isn't guaranteed to just work out of the box. Laptop WiFi cards, for example, frequently have driver issues. So it's not quite as simple as OP is saying, even with PCs.
Long(ish) answer: Each phone manufacture heavy modifies the linux android kernel in order to make it work on there hardware. This usually means that the binary blobs (there are a lot of them) are only compatible with specific kernels.
In addition, phone manufacturers write bad code, that would need to go through many review iterations until the open source community is happy with their changes. Often their changes to the kernel break things for other hardware, for instance.
Review and writing good software takes time, and phone manufactorers don't care about long term support, they want to sell you a new phone every year, or even more often. So they fork the kernel and hack together with lots of string and duct tape, a frankenstein kernel that just works on their hardware, but is broken for anything else.