We definitely make heavy use of passive Investigation--if your PI is higher than the save, you're going to realize it's an illusion as soon as your attention is drawn to the illusion as existing (so, one illusionary barrel in the middle of a storeroom filled with barrels that you're just glancing at--no, unless you've also got a really high passive Perception, but if you hear a noise and are looking for its source in that same room, yes). I think of it as being like something being drawn on an animation cel as opposed to the background.

The place things get a little tricky, to me, is if you are making an illusion of something that isn't solid. What if I make a Minor Illusion of a cloud of fog? "Physical interaction with the image reveals it to be an illusion, because things can pass through it" (emphasis mine)--things do pass right through fog, after all. If you touch the fog with your hand or a sword, I'll give you the free recognition, no question, but an arrow coming out of a fog bank isn't necessarily weird. Of course, you might expect the fog to swirl in response to said arrow...my compromise is to give anyone seeing it a free Investigation roll without having to choose to make it (with their PI as the floor for the roll, and if you do it more than once, they get advantage), but I could see declaring that lack-of-interaction enough for automatic detection.

OK, that's fog, where you do expect to see some physical interaction even if things pass through it. What about darkness, where you just make a sphere of blackness such as produced by the Darkness spell? I'd say this can't be done with Minor Illusion, where you actually have to make an object, but jumping up to Silent Image, you can make a "visual phenomenon." (If you want to argue that fog is not "an object", then consider the previous paragraph to also be about Silent Image.) People know you can just create clouds of fog or spheres of darkness, so having them just appear isn't a giveaway. My ruling is that arrows going in or out isn't enough to give you automatic detection, but I still have melee-range touching give it away. (I wouldn't argue with a DM who ruled either that arrows do give it away or that touching doesn't, though I think the latter makes it too powerful for a 1st level spell.)

This comes up a lot in our primary game, as my character has both Minor Illusion and unlimited use of Silent Image--note that since Minor Illusion isn't concentration and can do sounds, you can use your action one turn to set up Silent Image, then use Minor Illusion on the next turn to make that Image not so Silent.