Hey all, following the work over the weekend we're now running Lemmy 0.19.4. please post any comments, questions, feedback or issues in this thread.

One of the major features added has been the ability to proxy third party images, which I've enabled. I'll be keeping a closer eye on our server utilisation to see how this goes...

This weekend I'll be working to upgrade AZ to lemmy 0.19.4, which requires changes to some other back end supporting systems.

Expect occasional errors/slowdowns, broken images etc.

Once complete, I'll be making further changes to enable/tweak some of the new features.

UPDATE: one of the back end component upgrades requires dumping and reimporting the entire lemmy database. This will require ~1 hour of total downtime for the site. I expect this to kick off tonight ~9pm Perth time.

UPDATE2: DB dump/re-import going to happen ~6pm Perth time, ie about 10 minutes from this edit.

UPDATGE3: we're back after the postgres upgrade. Next will be a brief outage for the lemmy upgrade itself... after I've had dinner 🙂

UPDATE34: We're on lemmy 0.19.4 now. I'll be looking at new features/settings and playing around with them.

Its been a little while since I posted stuff :)

CPU: !

Memory: !

Network: !

Storage: !

Cloudflare caching: !

Comments: Not much has changed in quite a while. I still have a cron-job running to restart Lemmy every day due to memory leaks, hopefully this improves with future updates. Outside of that, CPU, memory and network usage are fine. Object storage usage is growing steadily, but we're a long way from paying more than the monthly minimum Wasabi fee.

The upgrade to lemmy 0.19 has introduced some issues that are being investigated.. but we currently have no fixes for:

• thumbnails break after a time. Caused by memory exhaustion killing object storage processes.
• messages to/from lemmy instance not yet running 0.19 are not federating. I believe it requires bugfixes by the devs.

~~I've re-enabled 2 hourly lemmy restarts. Hopefully this will help with both issues, though it will result in a disruption to the site around every couple of hours.

When the hourly restarts are disabled I'll unpin this post. As any other issues are identified I'll post them here too. ~~

Update: I've disabled the 2 hourly restart after upgrading to 0.19.2... lets see how this goes...

Update2: no issues seen since the upgrade, looks to have resolved both the memory leak and the federation issues. Hooray :)

I'll be restarting AZ today for an update to lemmy 0.19.

Upgrade complete.

This is a major upgrade, so I expect there to be some issues. Strap in, enjoy the ride.

Expect:

• further restarts
• bugs
• slowdowns
• logouts
• 2FA being disabled
• possibly issues with images, upgrading pictrs to 0.4 at the same time

Hi all,

Looks like I'll be visiting Melbourne in winter 2024 for a wedding with my family. Any must-see tourist attractions you can suggest? We'll definitely be hitting up Melbourne zoo, and maybe Werribee.

Thinking I'll have to hire a car.. though I'd prefer to stay central and use public transport and Uber around when required.

I'm kicking off a storage upgrade on the server, expect it to proceed in the next 15 minutes and be back online shortly after.

Here we are, another month down.. still kicking 😀

As usual I’m stating full dollar figures for simplicity, but they’re all rounded/approximate to the dollar. I’m not an accountant and its close enough for our purposes.

Income

$98 AUD thanks to 13 generous donors

Expenses

$45 OVH server fees $10 ($~6 USD)Wasabi object storage $39 ($25 USD) Domain registration extended to 6th August 2025 = $94

Balance

+$486 carried forward from July +$98 income for August -$94 expenses paid in August = $490 current balance

Future

Baseline storage on the server is now at ~70% at the low point of the day, time permitting I'll upgrade shortly to ensure sufficient room for growth. Doubling the current storage will cost an additional ~$12 per month.

THANK YOU

to everyone that has contributed to the running costs of the site.

A very special thank you to @Nath@aussie.zone for everything he's doing here, both in public and behind the scenes. Without his help performing admin and moderator activities, this place would be a shambles. I've simply been too busy professionally and personally to pay as much attention as I need to. Thanks Nath, I (and I'm sure everyone else here) appreciate it 🤗

As always, if you have any questions please ask.

CPU:

!

Memory:

!

Network:

!

Storage:

!

Cloudflare caching:

!

Summary:

Not much to call out. The storage drop was due to purging a days worth of images, and clearing the entire object storage cache. When I have time, I'll upgrade the VPS to add storage.

Due to some disgusting behaviour, I've kicked off the process of deleting ALL images uploaded in the last day.

You will likely see broken images etc on aussie.zone for posts/comments during this period of time.

Apologies for the inconvenience, but this is pretty much the nightmare scenario for an instance. I'd rather nuke all images ever than to host such content.

CPU:

!

Memory:

!

Network:

! The one spike here is from a DB backup being uploaded to object storage, prior to the upgrade to 0.18.4.

Storage:

!

Still ok here. You can see the daily minimum free space increase as I tweak the local cache for object storage.

Cloudflare caching:

!

Summary:

Nothing of note here. Once baseline storage hits ~70% and I decreasing the object storage retention period is no longer worthwhile, I'll upgrade the server for more storage.

Sorry this is a little late, I've been busy with real life this week.

I'm stating full dollar figures for simplicity, but they're all rounded/approximate to the dollar. I'm not an accountant and its close enough for our purposes.

Income

$267 AUD thanks to 25 generous donors, it is very much appreciated.

Expenses

$34 OVH server fees. Lower than normal, as OVH seem to switch services to calendar based rather than anniversary based from the second month. The July billing period covered 8 July - 31 July, August will be 1-31. $40 Cloudflare. Due to huge volumes of egress traffic from Cloudflare in early July, I opted to pay for a month of premium in order to investigate. The cause was found and remediated. I've since dropped back to the free plan.

Balance

$193 surplus from July (Income - Expenses) $293 carried forward from June = $486 current balance

Future

Wasabi's trial period expired on the 8th of July, so the first bill is due this month.

Domain registration, I'll be looking to extend by another year and keep it so that we always have at least 12 months paid up.

Server storage, I expect we'll need to upgrade for additional storage this month. With this upgrade I'll consider pre-paying/committing to the server for a longer period of time. This provides both a discount, and certainty to everyone that the money is put to good use.

THANK YOU

..again, to our generous donors. The offer of an @aussie.zone email address redirect is still open to any donors.

As always, if you have any questions please ask.

CPU:

!

Memory:

!

Network:

!

Storage:

!

Cloudflare caching:

!

Summary:

My only call out this week is an uptick in storage consumption, seems to align with an increase in new user signups and general higher activity. I'm guessing this is due to the release of Sync for Lemmy.

CPU:

!

Memory:

!

Network:

!

Storage:

!

Cloudflare caching:

!

Summary:

All resource usage looking stable. Storage is the only one that is trending up, as expected.

Another week... another bunch of nerd graphs!

CPU:

! Not much to say here, pretty stable CPU usage wise.

Memory:

! The unusual memory growth appears to have been related to a minor Postgres configuration change I made last week, which was reverted on Thursday. Memory usage looking much more normal since.

Network:

! As with CPU usage, network traffic is looking stable.

Storage:

! Storage growth has normalized, now that we've hit an equilibrium point. Though I'll be tweaking the object storage cache retention to minimise object storage pulls.

Cloudflare caching:

! Still saving us a large volume of egress traffic. Will save even more if particular content goes viral.

Summary:

Resource utilisation on the server is looking great across the board. No skyrocketing usage as we saw initially. Storage is still looking like the first trigger for another server upgrade, but as it is now a gradual increase we'll have plenty of fore warning and its looking like this will be some time away.

Questions? 🤓

Any chance we could get a setting to toggle the display of avatars in comments?

Doh! Forgot earlier in the night, so here you are... technically Saturday.

CPU:

! The lemmy devs have made some major strides in improving performance recently, as you can see by the overall reduced CPU load.

Memory:

!

I need to figure out why swap is continuing to be used, when there is cache/buffer available to be used. But as you can see, the upgrade to 8GB of RAM is being put to good use.

Network:

!

The two large spikes here are from some backups being uploaded to object storage. Apart from that, traffic levels are fine.

Storage:

! A HUGE win here this week, turns out a huge portion of the database is data we don't need, and can be safely deleted pretty much any time. The large drop in storage on the 9th was from me manually deleting all but the most recent ~100k rows in the guilty table. Devs are aware of this issue, and are actively working on making DB storage more efficient. While a better fix is being worked on, I have a cronjob running every hour to delete all but the most recent 200k rows.

Cloudflare caching:

!

Cloudflare still saving us substantial egress traffic from the VPS, though no 14MB "icons" being grabbed thousands of times this week 😀

Summary:

All things considered, we're in a much better place today than a week ago. Storage is much less of a concern, and all other server resources are doing well... though I need to investigate swap usage.

Longer term it still looks as though storage will become the trigger for further upgrades. However storage growth will be much more slow and under our control. The recent upward trend is predominantly from locally cached images from object storage, which can be deleted at any time as required.

As usual, feel free to ask questions.

Is it possible for Voyager to display user avatars in the same way as the regular web page? Its how I recognize regulars in my communities, more so than by name.

Available here

!

Noting this here for my own reference, and any other lemmy server admins that don't happen to be database administrators by day 🙂

I am not a DBA, if I'm doing something bad/incorrect here... please post! Yes, I've reset the password and TOTP token on the example account below.

2FA flags are stored in the local_user table, however that does not show usernames. To find the person_id for the user account you want to disable 2FA for, you'll need to check the person table. I'll use my test account here as an example:

SELECT * from person where name = 'guineapig' and local = 't';

Giving: !

Note the number 781227, this is the person_id for this account on my instance. To confirm: SELECT * from local_user where person_id = '781227';

!

Yep, the 2FA string has the expected username in it. Now to disable 2FA on the account we need to NULL out both totp_2fa_url and totp_2fa_secret rows: UPDATE local_user SET totp_2fa_url = NULL WHERE person_id = 781227;

UPDATE local_user SET totp_2fa_secret = NULL WHERE person_id = 781227;

Should give output like this: !

And checking the local_user table again, both TOTP fields should be empty:

!