What is the party line on foods that have labels that say "may contain milk, eggs, etc..."?

Personally I will not eat anything with the may contain label. I know this just means it is made in the same factory but I was wondering if I have just been overly cautious. What do you all think? Do you avoid stuff like that or do you think it is fine?