First thing here: You should always train on the base SD 1.5 model (or tha appropriate SD base model if you don't want to use 1.5), doing this should make you Embedding work better with more models.

Other than that, it might be an issue with your training data (you need about 25-30 images with different zoom levels and angles) or your descriptions. It might be helpful to post one of those (maybe with a made up name and the image anonymized) or your initialization vector or vector count.

That said, most models have a huge bias towards female looking bodies and there isn't much that can be done about it right now. I think this will smooth out later...

Not much said her (sorry about that) but i hope I could provide some pointers..

Edit: One thing that came to mind: Maybe your descriptions are too accureate. You should put everything into the description that's shown but not you. Everything that's in the description will not be part of the embedding.

The descriptions should contain

• background
• zoom level
• lighting (if ununsual)
• pose (if not neutral)
• expression (if not neutral)
• clothing (if not your signature shirt)

The description should not contain

• age
• color of skin, hair or eyes
• read gender
• hairstyle