All I can say is you just got federated! Pay it forward or something altruistic like that.
Working Setup files, for my ARM64 Ubuntu host server. The postgres, lemmy, lemmy-ui, and pictrs containers all are on the lemmyinternal network only. The nginx:1-alpine container is in both networks. docker-compose.yml
spoiler
version: "3.3"
# JatNote = Note from Jattatak for working YML at this time (Jun8,2023)
networks:
# communication to web and clients
lemmyexternalproxy:
# communication between lemmy services
lemmyinternal:
driver: bridge
#JatNote: The Internal mode for this network is in the official doc, but is what broke my setup
# I left it out to fix it. I advise the same.
# internal: true
services:
proxy:
image: nginx:1-alpine
networks:
- lemmyinternal
- lemmyexternalproxy
ports:
# only ports facing any connection from outside
# JatNote: Ports mapped to nonsense to prevent colision with NGINX Proxy Manager
- 680:80
- 6443:443
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
# setup your certbot and letsencrypt config
- ./certbot:/var/www/certbot
- ./letsencrypt:/etc/letsencrypt/live
restart: always
depends_on:
- pictrs
- lemmy-ui
lemmy:
#JatNote: I am running on an ARM Ubuntu Virtual Server. Therefore, this is my image. I suggest using matching lemmy/lemmy-ui versions.
image: dessalines/lemmy:0.17.3-linux-arm64
hostname: lemmy
networks:
- lemmyinternal
restart: always
environment:
- RUST_LOG="warn,lemmy_server=info,lemmy_api=info,lemmy_api_common=info,lemmy_api_crud=info,lemmy_apub=info,lemmy_db_schema=info,lemmy_db_views=info,lemmy_db_views_actor=info,lemmy_db_views_moderator=info,lemmy_routes=info,lemmy_utils=info,lemmy_websocket=info"
volumes:
- ./lemmy.hjson:/config/config.hjson
depends_on:
- postgres
- pictrs
lemmy-ui:
#JatNote: Again, ARM based image
image: dessalines/lemmy-ui:0.17.3-linux-arm64
hostname: lemmy-ui
networks:
- lemmyinternal
environment:
# this needs to match the hostname defined in the lemmy service
- LEMMY_UI_LEMMY_INTERNAL_HOST=lemmy:8536
# set the outside hostname here
- LEMMY_UI_LEMMY_EXTERNAL_HOST=lemmy.bulwarkob.com:1236
- LEMMY_HTTPS=true
depends_on:
- lemmy
restart: always
pictrs:
image: asonix/pictrs
# this needs to match the pictrs url in lemmy.hjson
hostname: pictrs
networks:
- lemmyinternal
environment:
- PICTRS__API_KEY=API_KEY
user: 991:991
volumes:
- ./volumes/pictrs:/mnt
restart: always
postgres:
image: postgres:15-alpine
# this needs to match the database host in lemmy.hson
hostname: postgres
networks:
- lemmyinternal
environment:
- POSTGRES_USER=AUser
- POSTGRES_PASSWORD=APassword
- POSTGRES_DB=lemmy
volumes:
- ./volumes/postgres:/var/lib/postgresql/data
restart: always
lemmy.hjson:
spoiler
{
# for more info about the config, check out the documentation
# https://join-lemmy.org/docs/en/administration/configuration.html
# only few config options are covered in this example config
setup: {
# username for the admin user
admin_username: "AUser"
# password for the admin user
admin_password: "APassword"
# name of the site (can be changed later)
site_name: "Bulwark of Boredom"
}
opentelemetry_url: "http://otel:4317"
# the domain name of your instance (eg "lemmy.ml")
hostname: "lemmy.bulwarkob.com"
# address where lemmy should listen for incoming requests
bind: "0.0.0.0"
# port where lemmy should listen for incoming requests
port: 8536
# Whether the site is available over TLS. Needs to be true for federation to work.
# JatNote: I was advised that this is not necessary. It does work without it.
# tls_enabled: true
# pictrs host
pictrs: {
url: "http://pictrs:8080/"
# api_key: "API_KEY"
}
# settings related to the postgresql database
database: {
# name of the postgres database for lemmy
database: "lemmy"
# username to connect to postgres
user: "aUser"
# password to connect to postgres
password: "aPassword"
# host where postgres is running
host: "postgres"
# port where postgres can be accessed
port: 5432
# maximum number of active sql connections
pool_size: 5
}
}
The following nginx.conf is for the internal proxy, which is included in the docker-compose.yml This is entirely separate from Nginx-Proxy-Manager (NPM)
nginx.conf:
spoiler
worker_processes 1;
events {
worker_connections 1024;
}
http {
upstream lemmy {
# this needs to map to the lemmy (server) docker service hostname
server "lemmy:8536";
}
upstream lemmy-ui {
# this needs to map to the lemmy-ui docker service hostname
server "lemmy-ui:1234";
}
server {
# this is the port inside docker, not the public one yet
listen 80;
# change if needed, this is facing the public web
server_name localhost;
server_tokens off;
gzip on;
gzip_types text/css application/javascript image/svg+xml;
gzip_vary on;
# Upload limit, relevant for pictrs
client_max_body_size 20M;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
# frontend general requests
location / {
# distinguish between ui requests and backend
# don't change lemmy-ui or lemmy here, they refer to the upstream definitions on top
set $proxpass "http://lemmy-ui";
if ($http_accept = "application/activity+json") {
set $proxpass "http://lemmy";
}
if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
set $proxpass "http://lemmy";
}
if ($request_method = POST) {
set $proxpass "http://lemmy";
}
proxy_pass $proxpass;
rewrite ^(.+)/+$ $1 permanent;
# Send actual client IP upstream
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# backend
location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
proxy_pass "http://lemmy";
# proxy common stuff
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Send actual client IP upstream
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
The nginx-proxy-manager container only needs to be in the same container network as the internal nginx:1-alpine container from the stack.
You need to create a proxy host for http port 80 to the IP address of the internal nginx:1-alpine container on the lemmyexternalproxy network in docker. Include the websockets support option.
https://lemmy.bulwarkob.com/pictrs/image/55870601-fb24-4346-8a42-bb14bb90d9e8.png
Then, you can use the SSL tab to do your cert and such. NPM is free to work on other networks with other containers as well, as far as I know.
Im actually making my little write up right now. Will post under the root thread shortly
Yea Ill do a write up in a bit with everything I can share that helped. Ill post it under the original thread. Not sure if I can sticky things but Ill try.
It would seem it was called "Internal" as opposed to isolated on my Portainer. That appears to have been it though. I can get to other communities now. Still having disparity with posts and comments showing up, but Im hoping that will be something to update in time.
Ive noticed mine DOES have it when viewed from another instance, however when I view it natively, I do not have it. Stuff is weird.
I see my Pictrs appears to be the same as what you had sent over. Protainer network isolation does not appear to be in place. All are bridged networks, and I would assume access issues would be more encompassing if that were a direct correlation to the issue. Im still betting on User Error for configuration so far. Being myself, of course.
The differences I see are the otel link, and the TLS setting:
# Whether the site is available over TLS. Needs to be true for federation to work.
tls_enabled: true
I see you dont have it on there, which I would assume means you cant be federated? I have added the otel link and enabled the debug mode. Federation is already enabled and the instance is set to "ALL". Still no luck on this end. Same status, except now im not getting any log errors in the container logs (Viewed from Portainer).
Including this in case it is a possible issue: federation enabled, host is lemmy.bulwarkob.com
Starting http server at 0.0.0.0:8536
Hey, does your instance have the "UI: v0.XX.X" tag at the bottom of the page? Mine does not, only the BE tag. I see other successful instances DO have it. Wondering if somehow related.
Not yet. Based on my error logs, Im thinking some sort of websocket issue where operations are timing out trying to communicate with other instances. Unfortunately I am not well familiar with NGINX and proxies in gerenal (hence using GUI based NPM). Im hoping I can find a kind soul to basically walk with me through my setup and see if the issue can be pinpointed. Ill definitely post how it got working if/when I get there.
I believe I have the simple set up for the NPM reverse proxy. Just as you say, it points to the docker address of the lemmyInstance NGINX. I can get to my instance with HTTPS secured and good. Just getting errors when communicating to other instances. I can only imagine it is web socket related, but I am not familiar enough to look at the proxy configs and determine what might be wrong with it unfortunately. I might need to try and find someone to essentially look through it with me in real time.
spoiler
ERROR HTTP request{http.method=GET http.scheme="https" http.host=lemmy.bulwarkob.com http.target=/api/v3/ws otel.kind="server" request_id=67d75886-bf48-4444-a435-d98d8fc1e303 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: Request error: error sending request for url (https://lemmy.ml/.well-known/webfinger?resource=acct:asklemmy@lemmy.ml): operation timed out
0: lemmy_apub::fetcher::search::search_query_to_object_id
at crates/apub/src/fetcher/search.rs:17
1: lemmy_apub::api::resolve_object::perform
with self=ResolveObject { q: "!asklemmy@lemmy.ml", auth: Some(Sensitive) }
at crates/apub/src/api/resolve_object.rs:21
2: lemmy_server::root_span_builder::HTTP request
with http.method=GET http.scheme="https" http.host=lemmy.bulwarkob.com http.target=/api/v3/ws otel.kind="server" request_id=67d75886-bf48-4444-a435-d98d8fc1e303 http.status_code=101 otel.status_code="OK"
at src/root_span_builder.rs:16
spoiler
WARN Error encountered while processing the incoming HTTP request: lemmy_server::root_span_builder: Request error: error sending request for url (https://beehaw.org/u/Jattatak): operation timed out
0: lemmy_server::root_span_builder::HTTP request
with http.method=POST http.scheme="https" http.host=lemmy.bulwarkob.com http.target=/inbox otel.kind="server" request_id=f413d3e5-262a-4dac-bc2e-700b9a053954 http.status_code=400 otel.status_code="OK"
at src/root_span_builder.rs:16
LemmyError { message: None, inner: Request error: error sending request for url (https://beehaw.org/u/Jattatak): operation timed out
Caused by:
0: error sending request for url (https://beehaw.org/u/Jattatak): operation timed out
1: operation timed out, context: "SpanTrace" }
I see that the instance can be reached, and posts are shown, however comments are not. I have found in the official docs that there is a config snippet for a web facing reverse proxy. https://join-lemmy.org/docs/en/administration/troubleshooting.html https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/nginx.conf
And this config appears quite different from the "Install with Docker" config instructions: https://join-lemmy.org/docs/en/administration/install_docker.html
spoiler
worker_processes 1;
events {
worker_connections 1024;
}
http {
upstream lemmy {
# this needs to map to the lemmy (server) docker service hostname
server "lemmy:8536";
}
upstream lemmy-ui {
# this needs to map to the lemmy-ui docker service hostname
server "lemmy-ui:1234";
}
server {
# this is the port inside docker, not the public one yet
listen 80;
# change if needed, this is facing the public web
server_name localhost;
server_tokens off;
gzip on;
gzip_types text/css application/javascript image/svg+xml;
gzip_vary on;
# Upload limit, relevant for pictrs
client_max_body_size 20M;
add_header X-Frame-Options SAMEORIGIN;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
# frontend general requests
location / {
# distinguish between ui requests and backend
# don't change lemmy-ui or lemmy here, they refer to the upstream definitions on top
set $proxpass "http://lemmy-ui";
if ($http_accept = "application/activity+json") {
set $proxpass "http://lemmy";
}
if ($http_accept = "application/ld+json; profile=\"https://www.w3.org/ns/activitystreams\"") {
set $proxpass "http://lemmy";
}
if ($request_method = POST) {
set $proxpass "http://lemmy";
}
proxy_pass $proxpass;
rewrite ^(.+)/+$ $1 permanent;
# Send actual client IP upstream
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
# backend
location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
proxy_pass "http://lemmy";
# proxy common stuff
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Send actual client IP upstream
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
}
Do you know if I should expect to have TWO unique NGINX proxy instances (assuming I use NGINX)? One in-stack, and one separate for web facing reverse proxy? Or do I need a combination of the two configs into one instance?
I am going to see if I can get a caddy reverse proxy setup in the meantime and see how it performs given your configuration there.
Hey, if you still feel like helping out :D
Ive been through a boatload of changes today since earlier. Ive rebuilt using mostly the provided yml in the official guide, and after some tweaking, almost everything is working. The internal proxy is now working, and the containers are working amongst themselves fully as far as I can tell. I do not know how to setup a web facing reverse proxy in a way that works around the internal proxy already running (other than the already in place NPM). I turned the NPM back on, and was able to get it working to reach the site, however I cannot reach any other communities from within my site. I believe the reverse proxy NPM is just not set up right. Error message in lemmy:
spoiler
ERROR HTTP request{http.method=GET http.scheme="https" http.host=lemmy.bulwarkob.com http.target=/api/v3/ws otel.kind="server" request_id=69004ca6-7967-48c3-a4d2-583e961e34d3 http.status_code=101 otel.status_code="OK"}: lemmy_server::api_routes_websocket: couldnt_find_object: Request error: error sending request for url (https://midwest.social/.well-known/webfinger?resource=acct:projectzomboid@midwest.social): operation timed out
0: lemmy_apub::fetcher::search::search_query_to_object_id
at crates/apub/src/fetcher/search.rs:17
1: lemmy_apub::api::resolve_object::perform
with self=ResolveObject { q: "!projectzomboid@midwest.social", auth: Some(Sensitive) }
at crates/apub/src/api/resolve_object.rs:21
2: lemmy_server::root_span_builder::HTTP request
with http.method=GET http.scheme="https" http.host=lemmy.bulwarkob.com http.target=/api/v3/ws otel.kind="server" request_id=69004ca6-7967-48c3-a4d2-583e961e34d3 http.status_code=101 otel.status_code="OK"
at src/root_span_builder.rs:16
I would be happy to remove NPM from this stack if its not too difficult to get a correctly working reverse proxy set up. The documentation doesnt give much to work with in it.
I actually started with this tutorial a few days ago after failing the official guide. I followed it but was unable to get it running due to unexpected errors. Im guessing this tutorial is somewhat out of date. Ive made progress since using that guide though so I will see if I can pull any useful bits out of it later today and continue.
Worst case, I could also just ditch NPM if I can get another NGINX set up in a way that you might know how to do correctly.
I appreciate your patience and clear assistance.
conf.d/* has two configurations that appear to be some form of default. default.conf and production.conf. production.conf is only for the admin GUI.
default.conf:
Container has a volume set /lemmy/docker/nginx-proxy-manager/data:/data
I have those folders and more, and they DO seem to have the correct custom item.
Specifically, in the proxy_host folder I have a configuration for the proxy host I set up (1.conf) in the GUI:
spoiler
# ------------------------------------------------------------
# lemmy.bulwarkob.com
# ------------------------------------------------------------
server {
set $forward_scheme http;
set $server "172.24.0.5";
set $port 1234;
listen 80;
listen [::]:80;
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name lemmy.bulwarkob.com;
# Let's Encrypt SSL
include conf.d/include/letsencrypt-acme-challenge.conf;
include conf.d/include/ssl-ciphers.conf;
ssl_certificate /etc/letsencrypt/live/npm-1/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/npm-1/privkey.pem;
# Block Exploits
include conf.d/include/block-exploits.conf;
# Force SSL
include conf.d/include/force-ssl.conf;
access_log /data/logs/proxy-host-1_access.log proxy;
error_log /data/logs/proxy-host-1_error.log warn;
location / {
# Proxy!
include conf.d/include/proxy.conf;
}
# Custom
include /data/nginx/custom/server_proxy[.]conf;
}
The rest of the folders are empty:
Hello! I have been struggling through a few tutorials on getting a lemmy instance to work correctly when setup with Docker. I have it mostly done, but there are various issues each time that I do not have the knowledge to properly correct. I am familiar with Docker, and already have an Oracle VPS set up on ARM64 Ubuntu. I already have portainer and an NGINX proxy set up and working okay. I have an existing lemmy instance "running" but not quite working. My best guess here would be to have someone assist with setting up the docker-compose to work with current updates/settings, as well as the config.hjson.
TIA, and I cant wait to have my own entry into the fediverse working right!