This should be how it is naturally, but for some reason there's very little interest in expanding ActivityPub's client-to-server protocol. It's sad that Mastodon chose to make their own API independent of everything else.
Nonetheless, there is Fedilab, which supports most microblogging platforms. Kaiteki is the only app I'm aware of that actually aims to support all of them. It's still in early development, but it's really promising.
From what I could tell from the code when I browsed it (which isn't much), Kaiteki actually holds dictionaries for popular kinds of ActivityPub objects. So, in theory, it could one day support everything natively.
I haven't really gotten a chance to try it, but ThatOneCalculator posts about it frequently, and I trust him more with the Fediverse's future than Eugen tbh. Most of Eugen's decisions track with his belief that fedi must grow faster, at any cost. Meanwhile, Calc seems to actually care about sustainable growth, and maintaining a community that cares about the tech. Which is essential imo.
There's room for both viewpoints, but it really is damning that most Mastodon apps don't support software that doesn't implement the Mastodon API.
Last time I used it, Fedilab also supported PeerTube. The same developer also released a specific PeerTube app, though so I don't know if they split the video functionality out or left it in.