Our solar system mostly revolves around the sun on the same axis (apart from Pluto). Our galaxy does the same (along with other galaxies). Why? Gravity is linear?
Would it matter if we tried to escape the sun's gravity by going "up?"
For reference, I'm a regular guy who looked up the answer, so maybe someone else can get more in depth, but I'll offer my basic understanding.
The planets weren't just plopped down in a straight line, they are all chunks of space debris that flew off of bigger chunks of space debris.
If you covered a ball in paint and spun it REALLY fast in a box slightly bigger than the ball, you'd end up with a line of paint on the walls that lines up with the center of the ball.
The planets are like that paint, but gravity essentially "reaches out and ties a tether" to them that keeps them from going further away. And the whole time, EVERYTHING is spinning and floating further away from the point of the Big Bang.
The last bit about the big bang isn't really how it works to my understanding. The big bang is compared to an explosion, but its actually more like a balloon inflating, if you imagine the surface of the balloon as analogous to space. The galaxies don't all move away from some original center to the universe, new bits of space get "added" in between every bit of space, so that every object gets farther away from every other object. If you go backwards in time far enough, every point sees itself as being the center. At least, that's how I've seen it explained.