Seems a little late to be adding in BBEVs at this juncture, unless this a 2 cours, or the manga drastically changes course going forward.
My impression on this show is it's supposed to be a light relaxing show with the occasional "problem" to solve (i.e. the guy abusing his fairy, or the daughter of the guy who ran the town trying to kidnap him) so having this round table of mustache twirlers added this late is a bit odd. I'll stick with it and see where it's going, but I hope it doesn't shift tones to drastically.
The source material is quite long (20+ volumes of light novels) and I think there will be elements introduced that don't pay off during this run of the anime.