Moving to a smaller instance is usually a good idea. Do keep in mind that content from lemmy.world could still give errors from time to time since it tends to get overloaded.
lemmy.world is one of the biggest instances, so I wouldn't be surprised if the server struggles from time to time.
Because of how the fediverse works, switching between local, all and subscribed doesn't make a difference. You still access the non-local content through lemmy.world because it's cached there.