One thing that mastodon does is proxying all the media from the federated servers, lemmy does not do this.. (yet)
For example on this comment page there are 9 domains trying to connect directly to me according to ublock origin. I suggest blocking all third party requests on your instance using ublock origins advanced mode because the website works fine without them, it might be mostly avatars?