Just recently watched a video on learning japanese by immersion

They say you have to do some ground work by learning the alphabets, basic grammar and some common words using SRS flash cards. Afterwards immersion is supposed to mostly fill in the gaps (you should tolerate ambiguity and let your brain figure out new words from the context). The video mentions it takes multiple hours a day for years to get good, but it also works with passive listening while doing something else. However they also talk about adding new words you hear to your SRS, mimicking what you hear, interacting with native speakers, etc so there's definitely a lot more to it than just sitting down and listening to podcasts.