If the difference between electrons and positrons, or quarks and antiquarks, is having opposite charge, what's the difference between chargeless neutrinos and antineutrinos?

Maybe a helpful visualization is one of the precursors to quantum field theory, Dirac's sea.

The idea is that you can think of a particle as sitting on top of the surface of the "sea" while an anti particle is represented by a hole in the surface, large enough to fit one particle. When a particle encounters such a hole, it naturally drops down into it and settles there. This essentially "destroys" both the particle and the hole (the anti particle).

So essentially the opposite charge, spin, etc of a particle and anti particle are a consequence of their opposition in their fields, not the cause for the annihilation.

(Not a scientist, grains of salt and all that).