Requirements This is a bug report, and if not, please post to https://lemmy.ml/c/lemmy_support instead. Please check to see if this issue already exists. It's a single bug. Do not report multiple b...
[X] Please check to see if this issue already exists.
[X] It's a single bug. Do not report multiple bugs in one issue.
[X] It's a frontend issue, not a backend issue; Otherwise please create an issue on the backend repo instead.
Summary
Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a "Server error".
FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
at ClientRequest.emit (node:events:511:28)
at TLSSocket.socketErrorListener (node:_http_client:495:9)
at TLSSocket.emit (node:events:511:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED'
}
Steps to Reproduce
Update docker-compose.yml to 0.18.0
docker-compose down
docker-compose up
access site
Technical Details
Docker setup. After updating to the 0.18.0 images for both lemmy-ui and lemmy backend, the lemmy-ui logs show a pictrs error, and the main site returns a "Server error".
FetchError: request to https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp failed, reason: connect ECONNREFUSED 127.0.1.1:443
at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
at ClientRequest.emit (node:events:511:28)
at TLSSocket.socketErrorListener (node:_http_client:495:9)
at TLSSocket.emit (node:events:511:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'ECONNREFUSED',
code: 'ECONNREFUSED'
}
So if that call fails for any reason, it breaks the SSR render.
I think this should be changed to return a broken image icon, or a blank image, instead of aborting the entire SSR process. Otherwise, the pictrs service going down for any reason causes the entire UI to crash like this.
That goes for any other fetches for external data too. If it's not mission critical (and images aren't mission critical), the SSR would ideally continue the render.
When trying to access the image directly at https://SITE_URL_REDACTED/pictrs/image/a29da3fc-b6ce-4e59-82b0-1a9c94f8faed.webp, it works fine in a browser.
The issue appears to be that lemmy-ui is trying to load the image from 127.0.1.1:443.
Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.
That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.
Issue "fixed" by putting the lemmy-ui container on a network with access to the internet. It looks like lemmy-ui didn't previously need an internet connection on 0.17.x, but now it needs one. Probably because it used to use the internal docker network to get to pictrs, but now it's trying to resolve using the real public hostname?
In OP's case, it's going to a local IP instead, but it's using the wrong port. In both our cases, the URL it uses for pictrs is not calculated correctly for whatever reason.
The UI could fail to contact pictrs for any variety of reasons, but that shouldn't be fatal. It should just break images.
Previous comment text:
I get a very similar error but it's not quite the same:
FetchError: request to https://<my-instance>/pictrs/image/<uuid>.png failed, reason: getaddrinfo EAI_AGAIN <my-instance>
at ClientRequest.<anonymous> (/app/node_modules/node-fetch/lib/index.js:1505:11)
at ClientRequest.emit (node:events:511:28)
at TLSSocket.socketErrorListener (node:_http_client:495:9)
at TLSSocket.emit (node:events:511:28)
at emitErrorNT (node:internal/streams/destroy:151:8)
at emitErrorCloseNT (node:internal/streams/destroy:116:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
type: 'system',
errno: 'EAI_AGAIN',
code: 'EAI_AGAIN'
}
Just like OP, I can go to the URL that error is logging and it pulls the image just fine. No browser cache shenanigans, the request is being fully completed.
My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?
I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!
My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?
@ubergeek77
short term fix is to remove your site’s image icon.
db query:
update site set icon = null where id = 1;
then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.
My server is now completely inoperable and I can't downgrade to 0.17.4 due to the database upgrades. Anything I can do to not lose my data?
@ubergeek77
short term fix is to remove your site’s image icon.
db query: update site set icon = null where id = 1;
then at least your site will work again, sort of. It loads at least. I am not sure federation is working 100% though and the pictrs errors still persist in the logs.
For an instance set up with Ansible:
Connect with ssh and cd /srv/lemmy/<domain>.
Run docker-compose down and look for the name of your postgres container (should be something like lemmyml_postgres_1) as it shuts down.
Run docker-compose up -d and wait for all the containers to start back up.
Run docker exec -it <postgres-container> "bash" to connect to your postgres database container.
Run psql -U lemmy to connect to the database with the lemmy user.
Run update site set icon = null where id = 1; then \q then exit
Nice! Good thing I backed up my DB. I’ll try changing the network settings and restoring my DB.
That did not fix my instance. I had to manually set the site’s image icon to null in the database to get my site working again. There are still many issues in both lemmy-ui and lemmy be logs.
Is your "external" NGINX proxy running on the same host server as Lemmy?
I found a more permanent fix. The error seems to be caused by having the host name of your server set to the same thing as your lemmy instance domain-name. The easy and more permanent fix is to change the host name of your server!
Had the same problem when I upgraded to 0.18.0, this fixed it.
Thanks! I just left that there for posterity, I added an edit to the top of my comment about how I "fixed" this.
It was caused due to a network error, so to fix it I just gave the container internet access, even though it otherwise wouldn't need it. That fixed things immediately.
Still, network errors shouldn't be fatal. Especially not for an icon.
I can confirm that this issue is still present in Lemmy-ui 0.18.1-rc.2. Every single time the docker images restart I have to run the commands to delete my site logo or the site will be unreachable. I think this may be partially because my external NGINX proxy is on a different different host. When Lemmy-ui resolves the IP for my domain it is still using the local IP and port 443.
If believe the fix would be to use the port specified by the pictures server in the internal NGINX configuration. I'm not aware of how to do this. I will test if this can be accomplished by configuring the pictures server to listen on 443 in the internal configuration.
Then, I changed my host's server name to not match the domain name, and rebooted. Local images were still not working after a docker-compose down and docker-compose up -d --force-rebuild. I still couldn't upload images either, and the site banner was not working (broken image).
The final step was to revert the lemmy.hjson back to http://pictrs:8080 (from the public domain name) and restart again, and now images are working, and having an icon for my instance no longer breaks the site.
So, to clarify, what I did was:
Change my host name in /etc/hostname so it no longer matches my site's domain name
Update lemmy.hjson to set pictrs back to the original http://pictrs:8080
Then, I changed my host's server name to not match the domain name, and rebooted. Local images were still not working after a docker-compose down and docker-compose up -d --force-rebuild. I still couldn't upload images either, and the site banner was not working (broken image).
The final step was to revert the lemmy.hjson back to http://pictrs:8080 (from the public domain name) and restart again, and now images are working, and having an icon for my instance no longer breaks the site.
So, to clarify, what I did was:
1. Change my host name in `/etc/hostname` so it no longer matches my site's domain name
2. Update lemmy.hjson to set pictrs back to the original http://pictrs:8080
3. Reboot, because why not?
4. "Restart" all lemmy containers via down, then up
5. 👍
can you write the instructions to change the hostname?