My grandpa is allergic to garlic. I could probably be fine with cutting out most allergens but garlic? That draws the line there.
A downside I never really thought of... is it more like a severe peanut allergy or like some sort of lactose intolerance?