[Maintenance Update & Postmortem] Lemmy 0.19.6-beta.13 and "Toastify is awesome!" (which it is.)

Firstly, the maintenance for Pawb.Social services has been completed and we've upgraded Lemmy to 0.19.6-beta.13, and our Mastodon instances to v4.3.0.

Thanks for your patience.

and now, onto the "Toastify is awesome" (because it is. Totally.) postmortem:

Short summary: Crash did a stupid with the config and emails weren’t getting sent (oops). Emails should now be working again, and registration shouldn't require admin intervention.

Longer story:

Lemmy has several configuration options that need to be updated to correspond to the specific instance that's running it. For example, here's a snippet of Pawb.Social's config:

{
  database: {
    host: "the.best.db.url.internal"
    user: "pawbsocial_lemmy"
    database: "pawbsocial_lemmy"
    password: "pawbsocial_do_you_really_think_i_would_leak_this"
    pool_size: 128
  }
  hostname: "pawb.social"
  pictrs: {
    url: "http://pawb-social-pictrs:8080/"
    api_key: "sekrit"
  }
  email: {
    smtp_server: "email-smtp.us-east-2.amazonaws.com:587"
    smtp_login: "pls_send_emails"
    smtp_password: "jeff_im_begging_you"
    smtp_from_address: "Pawb.Social Lemmy <{{domain}}>"
    tls_type: "starttls"
  }
}

Can you spot the problem? (No, it's not my awful password, or the pleading I'm doing with AWS to send emails so y'all stop having email problems.)

Originally, Lemmy was deployed through docker-compose which uses variable replacement to change something like {{domain}} to pawb.social automatically. Crash swapped away from using docker-compose due to repeated issues with trying to deploy the Lemmy changes and opted for a more bare Docker configuration.

Can you guess what didn’t get changed? …Yeah, the {{domain}} part was never removed and replaced with pawb.social so Lemmy correctly caught the error and threw an exception. The unfortunate part is that the thrown exception caused the whole comment process to close the connection to the web proxy server which returned a 502 Bad Gateway, which gave the UI HTML instead of a JSON response. The response handler in the UI then tried to figure out what was wrong, failed to translate the HTML to JSON, and gave the default error message “Toastify is awesome!” (which it is. Objectively.)

HOPEFULLY. This does fix the issue. I can’t guarantee it, but I’ve been able to replicate it by replying to someone else’s comment (which usually triggers an email notification) and after making the change, it successfully sent off the comment with a happy green confirmation toast.