How do devs at Windows, macOS, Linux etc. upgrade their OS using their existing OS?

Oversimplified:

• You have your current OS, text editor, compiler.
• You write code of the new improved OS using your current OS, text editor.
• You compile the code (text file), compilation yields the new OS or the new kernel (binary file).
• You replace (overwrite) your current kernel by the new kernel (current OS by new OS). This is possible, because while the OS is running it is in RAM not touching the disk.
• You restart.
• BIOS loads the new OS from disk to RAM and executes it.
• tada.wav

More questions:

• How to update BIOS? Answer: The same.
• How the first OS, text editor and compiler were created? Answer: The same. Using more primitive OS, text editor and compiler each step into the past. At the beginning there were toggle switches, punch cards, punch ribbon strips or similar.

The same style of question would be: How to create a hammer if in order to create a hammer you need a hammer? How was the first hammer created? Answer: By more primitive hammer, or something that is no hammer, but almost works as a hammer.

For more info read about bootstrapping compilers. Or trusting the trust by Ken Thompson.

They use the old one to build the new one. It's the same deal with compilers: to build the new version, you use your existing C compiler to build the new version of the compiler, then you use that build of the new compiler to recompile the compiler again itself (so you get the latest codegen from it), which results in the final compiler.

So for Windows 11, they developed it on Windows 10.

For macOS, they use the previous generation of Mac to build the new build of macOS, then they boot it up on another Mac to test it out with the first Mac being the host for a debugger connection and whatnot. For new hardware, it's essentially the same deal: they made M1 Macs using the previous generation Intel Mac, and when the hardware is ready to build the new OS from the Intel Mac and boot it up on the M1 Mac and test it and so on.

It's called the bootstrap process, and it essentially goes all the way to punch cards and mainframes. The first assembler was made directly in machine code, then with the first assembler you can make a better assembler, and with the better assembler one could build the first C compiler, then with the first C compiler you can make a better and more complex C compiler, and with that you can make the first C++ compiler, and on and on. You just use whatever you already have and build the new one using it.

Same goes with CAD design: first it was on paper, then we had computers where we could write CAD software to make better computers that can run better CAD software that lets you build even better computers and then better CAD software.

What I don’t understand is how they upgrade their existing software they themselves work in, especially when it has completely new features the old one doesn’t have. I feel like this is similar to a person performing a brain surgery on himself.

Basically, they don't. You build the next version using only features that the current version supports. Although in case of OS development specifically, you don't use OS features, you use compiler features and those don't evolve the same way at all. But the process is the same: you implement the new features exclusively using what the current compiler supports, then once you do have that new compiler, you can change the code again to make use of those new features and build the new version of the compiler using that. And then you recompile it again for good measure.

See: https://rustc-dev-guide.rust-lang.org/building/bootstrapping.html

While I'm not experienced enough to explain the full development stack of an OS. Let me throw my two cents.

It typically goes by writing changes. If its superficial ones, like modern UI in Windows 11, then all they need to do is relaunch explorer/the app etc. Every time they make a change in the code, they then build and try it out.

If its a more internal change, deep into the OS. Typically written in C or another low level language. Then its easier to test the changes in a virtual machine, you write your code, compile, build. And then load it up in the virtual machine to see if the OS doesn't crash and burn.

Later, after it gets past quality control in the company, (but most often these versions sit in beta for a while to catch problems). It then gets put into the Update servers and rolled out in bulk for mass destribution.

Do note, updates don't need to include the entire OS. Just packages including the file changes as well as general update busywork.

PS: If anyone replies, feel free to correct me. Details may be sketchy but this is the short of it.

One thing that nobody else is touching on which I think is part of what you're asking is that it differs from brain surgery in that they don't need to work on the "live" version of the operating system that they are using to develop on. They might have another or multiple other computers that they install their in-development versions of the OS on, or use virtual machines (which is where a virtual computer is created inside of another computer running a completely different OS) to test their changes. Operating systems are also very large and made up of specific components, a single person might be working only on a very specific part of the OS at one time which they can swap into an existing version of the OS in order to test it, rather than having to have a half finished windows 11 on their PC.

Microsoft at least used to famously “eat your own dog food “ during development of new Windows versions. This means that the developers themselves had to install the latest build and then use that while continuing development. This could lead to longer delays when crucial parts didn’t work.

AFAIK this trend started with David Cutler (legendary developer) and the development of the original Windows NT. You still need a fair bit of building blocks developed individually before you have something large enough to assembly to a build.

In those days, they sat mostly on OS/2 while developing parts for the kernel and boot system etc. Then some specifically precarious items to roll into the build and have everyone start using was networking, file system, and graphical interface. Bugs here hampered everyone (like, how do you create a new build of windows with new parts when networking / sharing files doesn’t work?).

Today I’ll imagine more development is performed on VMs so that it’s much less noisy when stuff doesn’t work. I’m sure at lot of developers still willingly or not install the version they are working on. However, it will probably be a little later with more working parts, and not updating every day etc.