I can imagine a rocket full of plutonium and uranium rods, sitting above immense tanks of combustible liquid: what the odds of such a rocket exploding during ascent? What are the effects of solar and cosmic radiation energy over these rods (even if they're lead shielded), especially around Van Allen radiation belt? So many questions.
We've been launching nuclear reactors into space for decades (mostly RTGs) they're just much smaller. There isnt any chance of them exploding or anything when exposed to radiation, but yes the chance of the rocket failing, exploding and showering radioactive material over the ocean is why this has to be done incredibly carefully if it is done.
According to my searches, while a RTG uses radioactive material weighting in the scale of kilograms (average of 5 Kg across missions such as Voyager and Cassini), a nuclear power plant requires several tonnes worth of plutonium and enriched uranium. The minimal critical mass for plutonium is 10kg, the double of how many fuel RTGs hold (that's why RTGs don't blow while ascending and/or on space). It's a large difference of mass/weight between RTG fuel and rods for nuclear power plants. They'd need to carry the whole tonnes worth of radioactive material split across very small quantities (which would require a lot of lead walling and/or launches)