That’s not monitoring. That’s just looking.

This post is about the stack I run instead: Telegraf collecting metrics off OPNsense, InfluxDB storing them as a proper time series, and Grafana turning that history into dashboards I can actually trust. None of the three pieces is exotic — this is a genuinely standard combination, sometimes shortened to “TIG stack” minus the L (Telegraf, InfluxDB, Grafana) — but getting OPNsense specifically to feed into it cleanly took a bit more care than I expected, and there are a few decisions along the way that are easy to get wrong without realizing it for weeks.

Why bother with a separate stack at all

OPNsense’s own dashboard isn’t bad. For a quick “is the box currently on fire” check, it’s genuinely useful. But it has three structural limitations that no amount of squinting fixes:

1. No real history. The graphs reset when you reload the page, and even when they don’t, they’re not backed by a queryable time series — you can’t ask “what did WAN throughput look like at 2am three Tuesdays ago” and get an answer.
2. No cross-host correlation. If you’re running two OPNsense boxes — which, if you’ve read Never Lock Yourself Out, you know I do, one at home and one fronting the colo stack — the built-in dashboard has no concept of “show me both of these side by side.”
3. No alerting on the data itself, beyond OPNsense’s own fairly basic notification system. You can get an email when a service crashes, but you can’t easily say “page me if WAN throughput sustains over 80% of link capacity for five minutes” without something purpose-built for that kind of rule.

A proper time series database and a real dashboarding tool solve all three at once, and once it’s running, it becomes the natural home for metrics from everything else in the homelab too — not just OPNsense.

The architecture, in one sentence

OPNsense runs a Telegraf agent that collects local metrics and writes them, over the network, into InfluxDB; Grafana then queries InfluxDB and renders the result as dashboards. Three distinct jobs, three distinct tools, none of them doing more than one thing.

OPNsense
  │  os-telegraf plugin (collection agent)
  │  inputs: cpu, mem, disk, interface, netstat, ping, pf states, dhcp leases
  ▼
InfluxDB
  │  time series storage, retention policy, downsampling
  ▼
Grafana
  │  queries InfluxDB, renders panels, fires alerts
  ▼
You, looking at a dashboard instead of guessing

I run InfluxDB and Grafana as Docker containers on the same dmz-stack host that runs Caddy, Authelia, and the rest of the DMZ services I’ve written about before — not on OPNsense itself. OPNsense is a firewall first; I’d rather not load it up with a database and a dashboard server it doesn’t need to be running, and keeping the heavier services off the firewall box means a bad container doesn’t have any chance of taking the firewall down with it.

Step 1 — Installing Telegraf on OPNsense

OPNsense doesn’t ship Telegraf by default, but it’s a one-click plugin install: System → Firmware → Plugins, search for os-telegraf, install it. Once installed, configuration lives under Services → Telegraf, split across a few tabs that map directly onto Telegraf’s own concepts — inputs, outputs, and general settings.

This is worth pausing on, because it’s easy to install the plugin, leave every input checkbox at its default, point the output at InfluxDB, and call it done — and you’ll get something working, but it won’t be the metrics you actually want. Telegraf’s inputs aren’t “on” by default just because the plugin is installed; you choose what gets collected.

The inputs I actually enable:

• CPU — per-core usage, not just an aggregate average. Aggregate-only CPU graphs hide a single pegged core behind three idle ones.
• Mem — straightforward, but worth keeping even on a box with plenty of headroom, since a slow memory leak over weeks is exactly the kind of thing a dashboard catches and a quick glance at the live widget never will.
• Disk / DiskIO — especially relevant if you’re logging anything verbosely to local storage, which on a firewall, you usually are.
• Interface — per-NIC bytes in/out, errors, drops. This is the single most useful input for a firewall specifically; throughput history per interface is the whole reason most people start this project in the first place.
• Netstat — connection counts, useful for spotting something hammering the box with new connections faster than normal.
• Ping — Telegraf can actively ping a target and record latency/loss itself, which is a nice complement to passive interface stats; I point this at my upstream gateway and at the Miami VPS from the CGNAT tunnel post, so a degrading tunnel shows up as a graph trending the wrong way, not just a missed handshake notification.
• pf states / firewall — OPNsense’s Telegraf plugin exposes pf state table size and related firewall internals, which matters more than it sounds like it should; a state table creeping toward its configured maximum is a real, dashboard-worthy warning sign, not just trivia.

I leave most of the more exotic inputs off. Telegraf’s input list is genuinely huge — it supports things this box will never need — and every enabled input is one more thing writing data, one more thing potentially misbehaving, one more thing to reason about later. Enable what you’ll actually look at.

Step 2 — Pointing Telegraf at InfluxDB

This is the part where version matters more than the OPNsense UI makes obvious. InfluxDB 1.x and InfluxDB 2.x are different enough in their authentication and data model that the Telegraf output configuration looks meaningfully different depending on which one you’re running.

If you’re on InfluxDB 1.x (database/retention-policy model, closer to classic SQL-ish thinking), the output block is the simpler of the two:

toml

[[outputs.influxdb]]
  urls = ["http://192.168.50.53:8086"]
  database = "opnsense"
  username = "telegraf"
  password = "REDACTED"

If you’re on InfluxDB 2.x (organization/bucket/token model), it’s a different output plugin entirely — influxdb_v2, not influxdb — and authentication is a token, not a username/password pair:

toml

[[outputs.influxdb_v2]]
  urls = ["http://192.168.50.53:8086"]
  token = "REDACTED"
  organization = "homelab"
  bucket = "opnsense"

Mixing these up is the single most common way to get a Telegraf service that starts cleanly, logs no errors, and silently writes nothing anywhere — because you’ve configured the v1 plugin against a v2 server (or vice versa), and the connection just fails quietly enough that it’s easy to miss for a while. If your Grafana dashboards are stubbornly empty and Telegraf’s own logs look unremarkable, this mismatch is the first thing I’d check.

I run InfluxDB 2.x, mainly because it’s the actively developed line and the bucket/token model maps more cleanly onto “one bucket per logical source, one scoped token per writer” than the older database/user model did. A scoped write-only token for the OPNsense Telegraf agent, separate from whatever token Grafana uses to read, means a compromised OPNsense box can write garbage into one bucket but can’t touch anything else in the InfluxDB instance.

Step 3 — Wiring Grafana to InfluxDB

Once data is actually landing in InfluxDB — worth confirming with influx query or the InfluxDB UI directly before touching Grafana at all, for the same reason I’d test any pipeline stage in isolation before chaining the next one onto it — adding it as a Grafana data source is the easy part: Connections → Data sources → Add data source → InfluxDB, point it at the same URL, hand it the read token (not the OPNsense write token), and pick the query language.

This is the other place version matters: InfluxDB 1.x speaks InfluxQL, a SQL-like query language; InfluxDB 2.x’s native language is Flux, which is a genuinely different paradigm — pipeline-style, more like a functional data-processing language than a query language. Grafana supports both, but which one you’re writing depends entirely on which InfluxDB version is on the other end, and Flux in particular has a learning curve if you’ve never written it before. I’d budget more time for “learning to write a Flux query that actually does what I want” than for any other single step in this whole setup — it was, for me, the slowest part by a wide margin, slower than anything on the OPNsense or Telegraf side.

A basic Flux query for, say, WAN interface throughput over the last six hours looks like this:

flux

from(bucket: "opnsense")
  |> range(start: -6h)
  |> filter(fn: (r) => r._measurement == "net")
  |> filter(fn: (r) => r.interface == "igb0")
  |> filter(fn: (r) => r._field == "bytes_recv" or r._field == "bytes_sent")
  |> derivative(unit: 1s, nonNegative: true)

That derivative() call matters more than it looks like it should — Telegraf’s net input reports cumulative byte counters, not instantaneous throughput. Graph the raw counter and you get a relentlessly climbing line that tells you almost nothing useful at a glance; take the derivative and you get the actual bytes-per-second rate, which is the number you actually wanted on the dashboard in the first place. This tripped me up for longer than I’d like to admit — the data was “correct,” it just wasn’t shaped the way a useful graph needs it to be.

Step 4 — Building dashboards that earn their keep

It’s tempting to build one enormous dashboard with every metric on it. Resist that. The dashboards that actually get looked at regularly, in my experience, are the narrow ones:

• Firewall health — CPU, memory, pf state table usage, disk. The “is this box okay” dashboard, checked reflexively the way you’d check a car’s dashboard lights.
• Throughput — per-interface bytes in/out, derived to rate, split by interface (WAN, LAN, the WireGuard tunnel interface specifically gets its own panel, since tunnel throughput tells a different story than raw WAN throughput).
• Tunnel and ping health — the active ping checks against the Miami endpoint and upstream gateway, graphed as latency and packet loss over time, rather than just the binary “handshake or no handshake” signal a status page gives you.

Splitting things this way also makes Grafana’s alerting more useful, since an alert rule tied to a narrow, specific panel (“pf state table over 80% of max for 5 minutes”) is something you can actually reason about and act on, where an alert buried inside a sprawling everything-dashboard tends to get ignored the same way a cluttered notification feed does.

There’s a fourth dashboard worth calling out separately: security signal. Once GeoIP blocking, CrowdSec, and the fail2ban-to-OPNsense sync are all feeding aliases on the firewall, Telegraf’s pf-state and firewall inputs end up indirectly reflecting how much of that traffic is actually getting dropped at the edge — a rising baseline of blocked-packet counts over time is a much better “is the noise increasing” signal than scrolling through raw logs ever was. The same goes for the fail2ban export pipeline — the size of that banlist file is itself a number worth graphing over time, since a sudden jump is exactly the kind of thing a dashboard catches faster than a person remembering to go check.

What I’d tell past-me before starting this

A few things that would have saved real time:

• Decide your InfluxDB version before writing a single Telegraf config line. Going back and forth between influxdb and influxdb_v2 output plugins after the fact, because you started on one version and migrated, is more annoying than just picking deliberately up front.
• Enable inputs deliberately, not by leaving every default checked. A Telegraf agent dutifully collecting forty metrics you’ll never look at isn’t free — it’s still I/O, still storage, still cardinality in InfluxDB, for genuinely no benefit.
• Remember that counters need derivatives. Any “total bytes” or “total packets” style metric is almost always more useful as a rate than as a running total. If a graph looks like a staircase that only ever goes up, that’s usually the tell.
• Keep write and read tokens separate, the same network-of-least-privilege instinct that shows up everywhere else in this series — scoped API tokens for ACME DNS challenges, scoped tokens for OPNsense’s own REST API, and now scoped tokens for InfluxDB too. It’s the same lesson, applied again.
• Put Grafana behind Authelia. It’s a web UI with real operational visibility into your network — exactly the kind of thing that deserves the same Authelia SSO and real two-factor setup as anything else in the DMZ, not a standalone admin/admin login sitting on its own.

Where this leaves things

A dashboard that actually has memory now. I can look at three weeks of WAN throughput, correlate a CPU spike with a specific timestamp instead of a vague memory of “it felt slow yesterday,” and get paged by Grafana on a real threshold instead of relying on a script that only checks one binary condition. The OPNsense web UI is still there for day-to-day configuration, but it’s no longer doing double duty as the only window into the firewall’s history — which, in hindsight, was always more than a single page of live widgets was ever going to handle well.

Search engines know too much about you. Every query, every click, every “how do I fix this embarrassing error” 2am search — logged, profiled, monetized. If you’re already running a homelab with a reverse proxy and an auth layer, there’s no good reason to keep feeding that machine. Run your own.

This is the build: SearXNG as a self-hosted metasearch engine, Authelia sitting in front of it for real authentication, Caddy doing the reverse-proxy handoff, and the whole thing wired into both your desktop browser and Open WebUI as the default search backend. No third-party API keys, no query logs leaving your network, no ads.

The architecture

Browser / Open WebUI
        │
        ▼
   Cloudflare (DNS + proxy, optional)
        │
        ▼
   OPNsense / HAProxy (TLS termination)
        │
        ▼
      Caddy (reverse proxy, forward-auth)
       │         │
       ▼         ▼
   Authelia   SearXNG
   (auth)    (search engine)

Caddy fronts everything and asks Authelia “is this request authenticated?” before forwarding to SearXNG. Browse to the search domain, get bounced to a login page with 2FA. Once authenticated, the session cookie carries you through.

If the OPNsense/HAProxy layer in that diagram looks unfamiliar, Beyond the Tunnel covers exactly this piece — HAProxy, ACME, and VLAN isolation sitting behind a WireGuard tunnel, which is what actually gets traffic from “the internet” down to Caddy in the first place. And if you’re starting from zero on getting traffic in at all — no public IP, ISP behind CGNAT — Bypassing CGNAT with an OPNsense WireGuard Tunnel is the prerequisite underneath that.

The one wrinkle: Open WebUI’s web-search feature calls SearXNG’s JSON API directly, with no human in the loop to click through a login screen. We’ll handle that with a network-scoped bypass rule rather than leaving the whole instance open.

Step 1 — Add SearXNG to your Docker Compose stack

If you’ve already got Caddy and Authelia running together — covered in One Login to Rule Them All, where the same Caddy-fronts-Authelia pattern was first wired up for Open WebUI and Nextcloud — this is just one more service in the same file.

yaml

services:
  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    restart: unless-stopped
    networks: [dmz]
    environment:
      - SEARXNG_BASE_URL=https://search.yourdomain.com/
      - TZ=America/yourtimezone
    volumes:
      - ./searxng:/etc/searxng:rw
    cap_drop:
      - ALL
    cap_add:
      - CHOWN
      - SETGID
      - SETUID
    logging:
      driver: "json-file"
      options:
        max-size: "1m"
        max-file: "1"

A few things worth calling out:

• SEARXNG_BASE_URL is what SearXNG uses to generate every self-referencing link — the OpenSearch descriptor, redirects, form actions. Get this wrong and things will mostly work, then quietly break in specific places (more on that pain later).
• Dropped capabilities — SearXNG doesn’t need root-level Linux capabilities to run. Drop everything, add back only CHOWN/SETGID/SETUID for the entrypoint’s user-switching. Standard container hardening, costs nothing.
• Bounded logs — max-size/max-file keep the container’s logs from growing unbounded on a long-running homelab box nobody babysits daily.

Bring it up:

bash

docker compose up -d searxng

Step 2 — Wire Caddy to front it

Add a site block to your Caddyfile. If you’ve already got a forward-auth snippet defined for other internal services — this is the exact same forward_auth block from One Login to Rule Them All — reuse it rather than writing a new one:

caddyfile

(forward_auth_block) {
  forward_auth authelia:9091 {
    uri /api/verify?rd=https://auth.yourdomain.com
    copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
    header_up X-Forwarded-Proto https
    header_up X-Forwarded-Host {host}
  }
}

search.yourdomain.com:80 {
  import forward_auth_block
  log {
    output file /var/log/caddy/access.log
  }
  header Strict-Transport-Security "max-age=15552000; includeSubDomains"
  reverse_proxy searxng:8080 {
    header_up Host {host}
    header_up X-Forwarded-Proto https
  }
}

Notice this listens on port 80, not 443. If TLS is terminated upstream (OPNsense/HAProxy, Cloudflare) and forwards plain HTTP into your Docker host, Caddy doesn’t need to handle certificates at all — auto_https off at the top of the Caddyfile makes that explicit.

Reload, don’t just restart, when testing Caddyfile edits live:

bash

docker exec caddy caddy reload --config /etc/caddy/Caddyfile

restart runs the existing container with the config it started with. reload tells the running process to re-read the file. They are not the same thing, and conflating them cost me twenty minutes of confusion mid-build — a request kept slipping past auth entirely, and the running config simply hadn’t changed.

Step 3 — Authelia access control

In Authelia’s configuration.yml, add a rule for the new domain:

yaml

access_control:
  default_policy: deny
  rules:
    - domain: search.yourdomain.com
      policy: two_factor

Validate before you restart anything — it’s free insurance against a YAML typo taking down your auth layer:

bash

docker exec -it authelia authelia validate-config --config /config/configuration.yml
docker restart authelia

Authelia’s session store (SQLite or Redis, depending on your setup) persists across restarts, so this won’t force-logout anyone already authenticated elsewhere. If you haven’t set up 2FA itself yet — TOTP enrollment, real SMTP for notifications, and a hostname-vs-IP TLS gotcha that’s easy to walk straight into — Locking Down Authelia covers that foundational setup before any of this access-control layer matters.

At this point, visiting search.yourdomain.com in a browser should redirect you to Authelia’s login page, prompt for 2FA, and land you on SearXNG after authenticating.

Step 4 — The Open WebUI problem, and the fix

Open WebUI’s web-search integration hits SearXNG’s API directly:

GET https://search.yourdomain.com/search?format=json&q=...

No browser, no cookie jar, no way to click through a login page. If your forward-auth rule applies universally, every one of these calls gets rejected — and crucially, an unauthenticated SearXNG instance with no access control is worse, since anyone on the internet could query it freely. If you’re exposing anything publicly at all, it’s worth having the broader edge-hardening picture in place — Hardening Your Homelab’s Public Edge covers GeoIP blocking and CrowdSec, the layer that sits in front of everything else described here.

The right middle ground is a network-scoped bypass: Authelia skips the auth challenge only for requests originating from inside your trusted network, not for the whole world. In configuration.yml:

yaml

access_control:
  rules:
    - domain: search.yourdomain.com
      resources:
        - "^/search.*$"
      networks:
        - 192.168.1.0/24    # your LAN/VLAN — wherever Open WebUI's host lives
      policy: bypass

    - domain: search.yourdomain.com
      policy: two_factor

The networks key scopes the bypass to source IP, not URL path. That distinction matters: a path-based bypass with no network restriction means anyone hitting that path skips auth, which defeats the point of putting Authelia in front of an internet-facing service in the first place. Scoping by network means the bypass only ever fires for traffic that’s already inside your perimeter. It’s the same network-scoped-trust principle behind the anti-lockout rule in Never Lock Yourself Out — trusting a request because of where it comes from, not just what it claims to be.

In Open WebUI’s admin settings (Admin Panel → Settings → Web Search), point it at:

http://searxng:8080/search?format=json

— the internal Docker service name, not the public domain. Since Open WebUI and SearXNG share a Docker network, there’s no reason to route this traffic out through Cloudflare and back in. Faster, and one less thing that can break.

Step 5 — Set it as your desktop default

In Firefox: about:preferences#search → Search Shortcuts → Add.

Name: SearXNG
Search string: https://search.yourdomain.com/search?q=%s

Select it under Default Search Engine, and every address-bar search now routes through your own instance, behind your own auth, with zero telemetry leaving your network beyond the search itself.

One practical note: with full Authelia 2FA in front of this, a long session expiration is worth setting deliberately, or you’ll be re-authenticating constantly for what should be a casual address-bar search:

yaml

session:
  cookies:
    - name: authelia_session
      domain: yourdomain.com
      authelia_url: "https://auth.yourdomain.com"
      expiration: 90d
      inactivity: 90d

90 days with 2FA on login is a reasonable balance — the login event is still strong, it just doesn’t need to repeat constantly for a service you check daily.

Things that bit me, in case they bite you

Environment variable changes don’t apply on restart. I changed SEARXNG_BASE_URL in the compose file and ran docker compose restart searxng. Nothing changed — the container kept the environment it was originally created with. restart cycles the existing container; it does not recreate it. Editing an env var in docker-compose.yml needs:

bash

docker compose up -d searxng

up -d diffs the desired state against the running container and recreates it if anything — including environment — has changed. This is the single most common trap in this entire build, and I hit it twice in one session before it stuck.

A renamed domain needs the cert checked, even if you’re sure it’s covered. If you’re running a wildcard cert (*.yourdomain.com), a new subdomain is automatically covered — but it’s worth confirming with a direct check rather than assuming:

bash

echo | openssl s_client -connect search.yourdomain.com:443 -servername search.yourdomain.com 2>/dev/null \
  | openssl x509 -noout -subject -issuer

If the browser throws a cert error referencing a domain you didn’t type, don’t assume the server is wrong before checking — browser-side HSTS caching and stale security-policy state can produce a cert error pointing at a hostname that has nothing to do with what’s actually being served. Verify the live cert from the command line first; it settles the question in one command instead of an hour of guessing. This is the same family of hostname-vs-certificate confusion covered in more depth in Locking Down Authelia — worth a re-read if this bites you here too.

Long-running containers can carry stale internal DNS. If a container’s been up for days while your host’s network configuration shifted underneath it, Docker’s embedded resolver inside that container can be working from outdated upstream DNS servers — it has no reason to refresh unless the container restarts. The symptom is oddly specific: internal Docker-network traffic works fine, but anything needing to resolve a real internet or LAN hostname fails outright. The fix is just a restart of that specific container, not a config change.

Where this leaves you

A search engine you fully control, parked behind real authentication, reachable the same way from a browser tab or from your local LLM’s tool-calling. No query logs handed to a third party, no API quotas, no ads. It took a Caddyfile block, one Authelia access-control rule, and a Docker Compose service — most of an evening was actually spent chasing two completely unrelated DNS issues that had nothing to do with SearXNG itself, which feels like the real lesson of running anything self-hosted: the new thing usually works fine. It’s the old thing sitting quietly next to it that breaks.

If you’re banning the inevitable scanners and bots that find this instance anyway, From App Logs to Firewall Blocks closes the loop on the network side — taking fail2ban’s detections and turning them into OPNsense-level blocks before a retry ever reaches Caddy at all.

More from the homelab security series

This post sits at the end of a fairly continuous build, in case you’re reading it without the others:

1. Bypassing CGNAT with an OPNsense WireGuard Tunnel — getting traffic in at all, with no public IP
2. Beyond the Tunnel: HAProxy, ACME, and VLAN Isolation — deciding what’s exposed once it’s in
3. Hardening Your Homelab’s Public Edge — GeoIP and CrowdSec at the perimeter
4. One Login to Rule Them All — Authelia as SSO for Open WebUI and Nextcloud
5. Locking Down Authelia — real 2FA, real SMTP, and the TLS trap
6. From App Logs to Firewall Blocks — fail2ban detections becoming OPNsense WAN blocks
7. Never Lock Yourself Out — the safety net underneath all of the above
8. This post — SearXNG behind Authelia, the newest piece on top of that stack

Except a username-and-password-only Authelia instance is just a fancier login form sitting in front of the exact same single point of failure every other login form has: whatever password you picked. The entire reason to run a centralized auth layer in the first place is to get to do hard things — like enforcing real two-factor authentication — once, centrally, instead of bolting it onto every app individually. If you skip that part, you’ve built infrastructure for a feature you’re not actually using.

So this post is about closing that gap: turning on real TOTP-based two-factor, and wiring up the notification system Authelia needs to actually deliver codes, links, and alerts — which sounds like a footnote but turned into its own small adventure once I hit a TLS certificate quirk that’s worth flagging for anyone else who runs into it.

Why two-factor isn’t optional once you centralize auth

Once Authelia is gating Nextcloud, Open WebUI, and whatever else lives behind it, that one login becomes the single key to the entire DMZ. That’s the whole point — but it cuts both ways. A compromised password used to mean a compromised app. Now it means a compromised everything, all at once, through the front door I built specifically to make access more convenient. Centralizing authentication without centralizing strong authentication is just centralizing risk — and it’s the same reason the DMZ network itself is VLAN-isolated from the rest of the homelab in the first place: a single compromised layer shouldn’t mean a compromised everything else too.

Authelia supports TOTP out of the box, and enabling it for an access_control policy is a one-line change in configuration.yml:

yaml

access_control:
  rules:
    - domain: openwebui.yourdomain.com
      policy: two_factor
    - domain: nextcloud.yourdomain.com
      policy: two_factor

That’s it on the policy side — flip one_factor to two_factor for any domain you want gated, restart Authelia, and the access control layer now requires a second factor before granting a session. I deliberately left auth.yourdomain.com itself — Authelia’s own login portal — on a bypass policy, since that’s the page doing the authenticating in the first place; there’s nothing to gate behind itself.

The part that actually takes thought isn’t the policy line, though — it’s how the person setting up TOTP for the first time actually receives the QR code and registration link. Authelia’s identity-verification flow for first-time 2FA registration works by emailing you a link to confirm it’s really you before letting you scan a QR code and bind an authenticator app. No working notification channel, no registration. The login page rendering correctly was never actually proof the auth system was complete; it was proof exactly one feature — first-factor password checking — was complete.

Bootstrapping without email: the filesystem notifier

Here’s a chicken-and-egg problem I didn’t expect going in: you generally need a working SMTP setup to register 2FA, but I hadn’t actually wired up SMTP yet when I wanted to register my own TOTP for the first time. Authelia’s answer to this is a notifier mode that doesn’t try to send anything anywhere — it just writes the notification to a file on disk:

yaml

notifier:
  filesystem:
    filename: /config/notification.txt

With this in place, instead of an email landing in an inbox, the identity-verification link gets written straight into a text file inside the Authelia container. Reading it is just:

bash

docker exec authelia cat /data/notification.txt

Copy the link out, paste it into a browser, and the registration flow proceeds exactly as if a real email had arrived. This is a genuinely useful bootstrapping trick worth knowing about independent of whatever your final SMTP setup ends up being — any time Authelia needs to send something and you don’t yet trust (or haven’t yet built) the real delivery path, dropping back to the filesystem notifier lets you keep moving without blocking on infrastructure that isn’t ready yet.

Wiring up real SMTP, and the trap waiting inside it

Bootstrapping aside, a filesystem notifier obviously isn’t a long-term answer — it only works because I have shell access to the box, which defeats the entire purpose of a notification system the moment any other person needs to register their own 2FA. The real fix is pointing Authelia at an actual mailbox.

The notifier config for SMTP looks innocent enough:

yaml

notifier:
  smtp:
    address: 'submissions://mailserver.domain.com:465'
    username: 'mymail@domain.com'
    password: 'REDACTED'
    sender: "Authelia <mymail@maildomain.com>"
    identifier: 'yourdomain.com'
    startup_check_address: 'mymail@maildomain.com'
    tls:
      skip_verify: false
      minimum_version: 'TLS1.2'

The submissions:// scheme tells Authelia this is an implicit-TLS SMTP submission connection on port 465, rather than the older plaintext-then-STARTTLS dance on port 587. That part’s fine and fairly standard.

What actually tripped me up was something that, in hindsight, is an obvious consequence of how TLS works, but is exactly the kind of thing you don’t think about until it bites you: my first instinct was to point the address field at the mail server’s raw IP rather than its hostname, since I already had the IP handy and it felt like one less moving part — no DNS resolution step, no dependency on DNS staying healthy. Authelia started up, attempted the connection, and immediately failed with x509: cannot validate certificate for <IP> because it doesn't contain any IP SANs.

That error message, once you actually read it instead of just reflexively googling it, explains the whole problem. The mail server’s TLS certificate was issued for the hostname mailserver.domain.com. TLS certificates assert an identity — “I am this specific name” — and the client connecting is supposed to check that the name it asked to connect to matches the name on the certificate it received back. When I connected by raw IP, there was no hostname in the conversation for the certificate to match against; the certificate says “I’m mailserver.domain.com,” and the connection was nominally to 203.0.113.45 or whatever the IP happened to be, and those two things have no relationship as far as the TLS handshake is concerned, unless the certificate specifically lists that IP as an additional valid identity (an IP SAN, which this one didn’t have, because almost no certificates bother to).

This is a small, almost embarrassing thing to get wrong, and also a genuinely common trap, because using a raw IP feels like it should be strictly safer or simpler — fewer dependencies, no DNS lookup that could fail or get hijacked. For plaintext or non-validated connections, sure. The moment TLS certificate validation enters the picture, the hostname isn’t an implementation detail you can route around — it’s load-bearing. The certificate’s whole job is binding a cryptographic identity to a name, and if you don’t connect using that name, there’s no name for the certificate to authenticate against, full stop. (The same hostname-vs-certificate confusion shows up again, in a slightly different shape, when verifying a wildcard cert covers a new subdomain — worth a read if this kind of TLS mismatch interests you beyond just SMTP.)

The fix was simply using the hostname Authelia was always meant to use:

yaml

address: 'submissions://mailserver.domain.com:465'

Once that one piece changed, the connection succeeded cleanly, the startup check passed, and Authelia logged a clean Startup complete with no certificate complaints.

Confirming it actually works, end to end

Configuration that merely starts without errors and configuration that actually works are two different claims, and I’ve been burned often enough by the first one masquerading as the second that I always insist on testing the real flow before calling something done. With SMTP wired up correctly, I went back through the TOTP registration process one more time, this time as a genuine end-to-end test rather than the filesystem-notifier bootstrap — triggered the identity verification flow, and watched a real email land in the actual inbox a few seconds later, with a working confirmation link.

From there, registering TOTP itself is almost anticlimactic after all that setup: scan the QR code with an authenticator app, enter the six-digit code it generates to confirm the binding succeeded, and from that point forward, every login to a two_factor-policy domain prompts for that code after the password. The login page, from the outside, looks exactly like it did before any of this work — username, password, and now an extra six digits. The difference is everything behind that screen actually does what centralizing authentication was supposed to buy in the first place.

The actual lesson here

None of this was hard, exactly — every individual piece was a short config block and a restart. What made it worth writing up is how each step quietly assumed the previous one actually worked, and how easy it would have been to stop at “the login page renders” and call it done. Two-factor without a working notification channel isn’t two-factor, it’s a setting nobody can finish enabling. A notification channel that starts without errors isn’t necessarily a notification channel that delivers anything, if a hostname-vs-IP TLS mismatch is quietly swallowing every send attempt. The only way to know any of it actually works is to walk the whole path a real, first-time user would take, end to end, and watch what actually lands in an actual inbox.

This same 2FA-gated access_control pattern gets reused directly in the next build — Ditch Google, which gates a self-hosted SearXNG instance behind exactly this TOTP setup, plus a network-scoped bypass rule for Open WebUI’s API calls. And if locking yourself out of your own auth layer is a worry — it should be, once you’ve made Authelia the single key to everything — Never Lock Yourself Out covers the equivalent safety net on the OPNsense side.

Two firewalls, two very different jobs. But they share a problem that took me longer than I’d like to admit to solve properly: how do you restrict management access on the remote box to “just me, at home” when your home connection doesn’t have a fixed IP — and worse, doesn’t even have a real IP from the internet’s point of view?

This post walks through what I built: a self-updating allowlist that tracks my home IP automatically, even through CGNAT, without me touching either firewall again. If you’re running a similar two-site setup — homelab plus colo, or homelab plus VPS — this is the kind of plumbing that’s easy to skip and annoying to retrofit later.

Why Bother Locking This Down At All

The webhosting firewall has a management GUI and SSH, like every OPNsense box does. In an ideal world, neither would ever touch the WAN interface — everything would go over VPN, full stop. In practice, there are legitimate reasons to want a direct fallback path: VPN daemons crash, tunnels flap, and at 2am when something’s on fire, you want a door that doesn’t depend on the thing that might be broken.

The compromise I landed on is the same one most of us reach for eventually: keep the door, but restrict who can knock. An allow rule scoped to “traffic from my home IP only” on top of a default-deny stance shrinks the attack surface from “anyone on the internet” to “anyone who happens to be sitting at my house, or who happens to share my ISP’s NAT pool at this exact moment” — which, as I’ll get into, isn’t quite as airtight as it sounds, but it’s a meaningful improvement over leaving the GUI wide open.

This is also just good practice independent of the CGNAT wrinkle. If you’ve got a static IP at home, the IP-allowlist part of this is trivial — one alias, one rule, done. The interesting part of my setup, and the reason this post exists, is what happens when your home connection doesn’t give you a stable address to allowlist in the first place.

The CGNAT Catch

My home ISP, like most residential providers these days, doesn’t hand out a unique public IPv4 address per customer. Instead, I sit behind Carrier-Grade NAT: a whole pool of subscribers shares a smaller set of public IPs, and the address my OPNsense’s WAN interface reports is a private one in the 100.64.0.0/10 range — not my actual internet-facing address at all.

This is the same CGNAT wrinkle that made getting a WireGuard tunnel through to this homelab in the first place its own separate problem — there, the issue was inbound traffic having nowhere to land; here, it’s an outbound-facing allowlist having nothing stable to allowlist. Same root cause, two different symptoms.

This matters a lot for dynamic DNS. The naive approach — point a DDNS client at your WAN interface and have it push that address to a DNS record — completely falls apart under CGNAT. You’d faithfully update your DNS record every few minutes with an address nobody on the internet can route to. The fix isn’t complicated once you know it’s needed, but it’s a “you don’t know what you don’t know” trap: the DDNS client has to ask an external service what your IP looks like from the outside, rather than trusting what your own interface thinks it is.

Building a Self-Updating Allowlist

With that constraint in mind, the chain I built has three links:

1. A DNS record on Cloudflare that always points at my real home IP.
2. A DDNS client on the home OPNsense box that keeps that record current — sourcing the IP from an external lookup service instead of the WAN interface, specifically because of the CGNAT issue above.
3. A firewall alias on the webhosting OPNsense box that resolves that same hostname on a timer and feeds the result straight into the pf table, with a rule referencing the alias instead of a hardcoded address.

The elegant part is that none of this needs glue scripts or cron jobs duct-taped together. OPNsense’s alias system natively understands FQDNs in Host-type aliases — it resolves them on a configurable interval (300 seconds by default) and keeps the underlying table in sync on its own. Once the chain is wired up, it just runs.

Step 1 — A DNS Record and a Properly Scoped API Token

On the Cloudflare side, I added a plain A record — home.yourdomain.com — with the proxy turned off (grey cloud, “DNS only”). That last bit matters more than it looks: if the orange-cloud proxy is on, anything resolving that hostname gets back one of Cloudflare’s edge IPs instead of my home address, which would quietly break the whole scheme in a way that’s annoying to debug.

For authentication, I skipped the Global API Key — it grants account-wide access, which is a lot of blast radius for something that only needs to touch one DNS record — in favor of a scoped API token limited to Zone / DNS / Edit permissions on the specific zone. Good habit generally: minimum necessary privilege, even for the small stuff.

Step 2 — DDNS on the Home Firewall

OPNsense doesn’t ship dynamic DNS support out of the box; it’s a plugin (os-ddclient). Once installed, the configuration lives under Services → Dynamic DNS, and the account setup is where the CGNAT-specific decision actually gets made.

The provider is Cloudflare, the token goes in the password field (with a slightly odd quirk — the username field has to literally contain the word “token” rather than an email address, which is how the plugin distinguishes API-token auth from the legacy global-key auth). The zone is the bare domain, the hostname is the full subdomain I want updated.

The setting that actually solves the CGNAT problem is “Check IP method.” Left on “Interface,” the client would read the WAN interface’s address — the private CGNAT one — and dutifully push garbage upstream. Pointed instead at a web-based IP-checking service, the client asks an external host what address the request arrived from, which is the real, routable, internet-facing IP, regardless of how many layers of NAT sit between my router and the actual internet.

It’s a small dropdown selection with a big practical consequence, and it’s the kind of detail that’s invisible until you’ve had a DDNS setup silently failing for a week because nobody told you the interface address wasn’t the address that mattered.

Step 3 — Alias and Rule on the Webhosting Firewall

On the colo box, the configuration is almost anticlimactic by comparison. A new Host-type alias with the DDNS hostname as its content, and OPNsense takes care of the rest — resolving it on its own schedule and populating the pf table transparently. Any rule referencing that alias automatically tracks whatever the hostname currently resolves to.

The rule itself is a single allow entry, sourced from the alias, scoped to the specific port the management interface listens on, sitting above the default-deny rule that covers everything else. From the rule’s perspective, it looks exactly like a static IP allowlist. The dynamic part is entirely hidden inside the alias. (This same idea — trusting a request because of where it’s coming from, rather than what it claims to be — is exactly what the network-scoped Authelia bypass rule for Open WebUI’s search calls does at the application layer instead of the firewall layer.)

What Could Still Go Wrong

I want to be honest about the limits of this setup, because IP-based access control always sounds more solid than it actually is.

First, there’s a convergence gap. If my home IP changes, there’s a window — typically a few minutes — between the DDNS client noticing and pushing the update, and the remote firewall’s alias re-resolving the hostname. During that window, the old IP is still allowed and the new one isn’t yet. If the only path into the box were this rule, a badly timed IP rotation could lock me out at the worst possible moment.

Second, and more fundamentally: CGNAT means the public IP I’m allowlisting isn’t uniquely mine. It’s shared across a pool of subscribers on the same ISP infrastructure. Anyone else sitting behind the same NAT pool at the same moment would also satisfy the source-IP check. That doesn’t make the rule useless — it still meaningfully narrows things down from “the entire internet” to “people on my ISP’s network right now” — but it’s not the same security guarantee a unique static IP would give you, and I don’t want to oversell it as one.

Because of both of these, this rule isn’t my primary path into the box. I already run a WireGuard tunnel between the two sites, with HAProxy, ACME, and VLAN isolation deciding what’s actually reachable on the other end of it, and that’s where day-to-day management traffic goes. This dynamic allowlist exists as the fallback — the door I can still use if the tunnel itself is down — not as the front line. And I’ve kept a separate, non-network access path (console/out-of-band) available too, specifically so that a mistimed IP change during a real outage can’t compound into a full lockout.

The Bigger Lesson

None of this is exotic infrastructure — a DNS record, a small plugin, two firewall objects. But it’s a good small-scale example of something that shows up constantly in the day job too: a control that looks complete in isolation often has a hidden dependency that only surfaces under specific conditions. “Just allowlist the IP” is a perfectly fine instruction right up until someone’s home connection sits behind carrier-grade NAT, at which point the naive implementation fails silently instead of loudly.

The fix, in both the homelab and the enterprise version of this problem, is the same: don’t treat any single control as sufficient on its own. Layer the dynamic allowlist behind the VPN, keep an out-of-band path alive, and build in the assumption that any one piece — DNS propagation, a NAT pool reassignment, a tunnel daemon crashing — will eventually fail at an inconvenient time. The goal isn’t a system that never breaks. It’s one where a single break doesn’t turn into a 2am lockout. The same defense-in-depth instinct shows up again later in locking down Authelia’s own 2FA setup — never trusting that one layer rendering correctly means the whole chain behind it actually works.

That’s the version of “secure by default” I actually trust — not because it’s clever, but because it assumes its own cleverness will eventually be wrong about something.

For a long time, my fail2ban setup did exactly what fail2ban is supposed to do, and I was perfectly happy with it. Watch the logs, count the failures, ban the IP at the host firewall level via nftables. WordPress brute-forcers, badbots scraping for wp-login.php, the usual cast of characters — all of them got banned, all on the backend server itself.

The problem with that picture only became obvious once I actually thought about where the ban was happening. My backend, the box running Apache and ISPConfig and all the actual application logic, sits behind OPNsense and HAProxy, with a WireGuard tunnel carrying the traffic between the two boxes in the first place. Every single request that gets banned by fail2ban has already traveled all the way from the internet, through OPNsense, through HAProxy, before fail2ban even gets a chance to say no. The ban was real, but it was the last possible line of defense, not the first. Every blocked request still cost me a hop through the entire stack before being rejected.

That’s the itch this post scratches: take the bans fail2ban is already correctly deciding on, and push them out to the actual edge — OPNsense itself — so the next attempt from that IP never even reaches HAProxy, let alone the backend. The attacker gets stopped at the door instead of in the living room.

Why this isn’t a five-minute job

The annoying part of this problem is that fail2ban and OPNsense don’t speak the same language. fail2ban lives on a Linux box and bans IPs using nftables or iptables rules, scoped entirely to that machine. OPNsense is a separate firewall appliance running pf, with its own rule engine and its own way of thinking about IP lists, called aliases. There’s no built-in bridge between the two. fail2ban has no idea OPNsense exists, and OPNsense has no idea fail2ban exists. If you want one to inform the other, you have to build that bridge yourself.

OPNsense, fortunately, gives you a reasonable on-ramp: a URL Table (IPs) alias type, found under Firewall → Aliases. Instead of manually typing a static list of IPs into an alias, you point this alias type at a URL, and OPNsense periodically fetches that URL, expecting a plain text file with one IP per line, and uses the contents as a dynamically updating block list. That’s the missing piece — if I can get fail2ban to maintain a text file with all its currently-banned IPs, and serve that file somewhere OPNsense can reach, the rest is just OPNsense doing what it already knows how to do.

So the pipeline has three parts: a script that asks fail2ban “who’s currently banned, across every jail you’re running” and writes that to a file; a way to serve that file so OPNsense can fetch it; and an OPNsense-side alias and firewall rule that actually uses the list to block traffic at the WAN edge.

Part one: asking fail2ban what it knows

fail2ban already tracks every jail’s banned IPs internally — you can query any individual jail with fail2ban-client status <jailname>, and it’ll show you a Banned IP list line. The annoying bit is that if you’re running several jails (and most reasonably hardened setups are — separate jails for WordPress login attempts, badbots, 404 floods, and so on), you need to loop over all of them and merge the results, deduplicated, into one list.

bash

#!/bin/bash
OUTPUT_FILE="/home/user/dmz-stack/caddy/banlist/banned.txt"
TMP_FILE="$(mktemp)"

for jail in $(fail2ban-client status | grep "Jail list" | sed 's/^.*://;s/,//g'); do
    fail2ban-client status "$jail" | grep "Banned IP list" | sed 's/^.*://' | tr ' ' '\n'
done | grep -v '^$' | sort -u > "$TMP_FILE"

mv "$TMP_FILE" "$OUTPUT_FILE"
chmod 644 "$OUTPUT_FILE"

This is deliberately unfancy. fail2ban-client status with no jail name lists every active jail; the loop asks each one for its banned IPs, strips the label text fail2ban prepends, and splits the space-separated list onto individual lines. Sorting and deduplicating means it doesn’t matter if the same IP shows up in two jails — it lands in the output file exactly once. The mktemp-then-mv pattern matters more than it looks like it should: writing directly to the final file risks OPNsense fetching a half-written file mid-update, while writing to a temp file and atomically moving it into place guarantees whoever reads banned.txt always sees a complete, consistent snapshot.

Drop this in /usr/local/bin/export-fail2ban-bans.sh, make it executable, and you’ve got the first link in the chain. Run it manually once and cat the output file to sanity-check it’s actually populated with real-looking IPs before moving on.

Part two: serving the file without opening a new can of worms

Now OPNsense needs to fetch this file over HTTP. The lazy option is to drop it somewhere your existing web server already serves — but I didn’t want a list of banned IPs sitting in a publicly browsable directory, even an obscure one, so I gave it its own dedicated, narrowly scoped route instead.

Since I already run Caddy as my reverse proxy in front of everything else, adding one more tightly restricted route was the path of least resistance:

caddyfile

:8090 {
  @opnsense remote_ip 172.18.32.1
  handle @opnsense {
    root * /srv/banlist
    file_server
  }
  handle {
    abort
  }
}

That remote_ip matcher is doing the actual security work here — only requests originating from OPNsense’s own WireGuard IP get served anything; everyone else hits abort, which just drops the connection outright rather than politely saying “no” with an HTTP error (politely saying no still tells a port-scanner something is listening). Putting this on its own port, 8090, separate from your normal 80/443 traffic, also means a casual look at access logs for your real sites never gets cluttered with OPNsense’s periodic polling requests. Mount the directory containing banned.txt into the container, expose the port in your compose file, and this half is done.

Part three: getting OPNsense to actually act on it

Inside OPNsense’s web UI, go to Firewall → Aliases and create a new alias. Set the type to URL Table (IPs), point the content field at the internal address of wherever you’re serving the file from — in my case http://172.18.32.11:8090/banned.txt, reachable over WireGuard — and set a refresh frequency. I run mine on a 5-minute cadence, which, combined with fail2ban’s own cron job re-running the export script every five minutes too, means worst-case latency between a ban happening and OPNsense enforcing it at the edge is comfortably under ten minutes. That’s not instant, but it’s a different category of protection than “eventually gets blocked after wasting a hop through my whole stack every single time.”

Once the alias exists and OPNsense has successfully fetched it at least once (you can confirm this under the alias’s own details view, which shows the current resolved IP count), the last step is actually using it. Go to Firewall → Rules → WAN, add a new rule with action Block, source set to your new alias, and — this part actually matters — drag the rule to the very top of the WAN rule list. Firewall rules are evaluated top-to-bottom, first match wins, and if some broader “allow” rule sits above your block rule, the block never gets a chance to fire.

Testing it without waiting for a real attacker

I didn’t want to wait around for an actual bot to get banned just to confirm the whole pipeline worked end to end, so I manually banned a couple of made-up test IPs directly through fail2ban (fail2ban-client set <jail> banip 203.0.113.99), ran the export script by hand, and watched the alias refresh in the OPNsense UI to confirm the count ticked up and the test IP appeared in the resolved list. Unbanning the same way and re-running the export confirmed it cleared correctly too. This kind of synthetic round-trip test is worth doing deliberately rather than trusting the pipeline blind — it’s much easier to debug a test IP that you know should or shouldn’t be there than to debug a real attacker’s IP weeks later while also wondering whether the pipeline ever worked at all.

What this actually buys you

It’s tempting to think of this as a marginal optimization — fail2ban was already blocking these IPs, so who cares if the block happens one hop earlier? But the difference compounds in ways that aren’t obvious until you’ve lived with it. Every blocked request that used to traverse OPNsense, HAProxy, and reach the application layer before being rejected was still consuming connection slots, CPU cycles, and log lines at every layer along the way. Pushing the block to the WAN edge means a banned attacker’s retry traffic gets silently dropped before it ever touches HAProxy’s connection table, which matters more than you’d expect once you’re also running Suricata and CrowdSec at that same edge — fewer packets reaching the inspection layers means cleaner signal in those tools too, since they’re not wading through traffic that’s already a known, settled case. Hardening Your Homelab’s Public Edge covers that GeoIP/CrowdSec layer in full — worth noting that post describes a different fail2ban-to-OPNsense sync method (a direct API push from fail2ban itself) than the pull-based, flat-file approach built here, so treat them as two alternative implementations rather than the same pipeline described twice.

There’s also a quieter benefit: once this pipeline exists, it becomes the natural place to plug in anything else that produces a list of bad IPs. CrowdSec’s own community blocklist already feeds OPNsense through a similar mechanism; Suricata’s alert-driven blocks could, in principle, feed the same kind of alias. The pattern — local detection, exported as a flat list, consumed by the edge firewall as a dynamic alias — turns out to be a reusable shape for tying together security tools that were never designed to talk to each other directly. fail2ban doesn’t need to know OPNsense exists, and OPNsense doesn’t need to know fail2ban exists. They just both need to agree on the shape of a text file, and that turns out to be enough.

This same “trust based on network origin, not credentials” idea reappears in two later posts: the network-scoped Authelia bypass for Open WebUI’s search calls, and the self-updating anti-lockout alias for remote firewall management — both are the same alias-driven, dynamically-resolved trust pattern used here, just applied to allowing traffic instead of blocking it.

Under the hood, one of my apps has no idea Authelia exists. The other one talks to it directly, as a first-class citizen, using a protocol with its own handshake, its own tokens, its own opinions about what a “user” even is. Get this distinction wrong — or worse, not realize it’s a distinction at all — and you’ll spend an evening debugging a problem that looks like a bug but is actually a category error. I know this because I did exactly that a few weeks later, but that’s a story for a different post.

For today: how do you actually wire Authelia in front of Open WebUI and Nextcloud, and why are these two setups fundamentally different shapes, not just different flavors of the same thing?

The app that doesn’t know it’s protected

Open WebUI, like a lot of self-hosted tools, was never built with an external identity provider in mind. It has its own login form, its own user table, its own idea of “logged in.” If I want Authelia sitting in front of it, I’m not integrating with Open WebUI’s authentication — I’m overriding it from the outside, before any of its own login logic ever runs.

This is forward-auth, and the mental model is simpler than the name suggests: Caddy intercepts every request to openwebui.yourdomain.com and, before it reverse-proxies anything to Open WebUI, makes a side-channel call to Authelia asking one question — “is this person allowed in?” If Authelia says yes, Caddy lets the request through to Open WebUI exactly as if nothing happened. If Authelia says no, the visitor gets redirected to a login page, and Open WebUI never even sees the request.

The forward-auth block in my Caddyfile looks roughly like this:

caddyfile

(forward_auth_block) {
  forward_auth authelia:9091 {
    uri /api/verify?rd=https://authelia.yourdomain.com
    copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
  }
}

openwebui.yourdomain.com:80 {
  import forward_auth_block
  reverse_proxy openwebui:8080
}

Two things matter here that are easy to skim past. First, copy_headers — once Authelia approves the request, it hands back a set of headers identifying who just logged in, and Caddy stitches those onto the request before it reaches Open WebUI. Second, and this is the part that actually matters for day-to-day reliability: Open WebUI still has no idea any of this happened. It sees a request arrive, optionally with some Remote-User-style headers it may or may not even read, and that’s it. As far as Open WebUI’s own code is concerned, Authelia doesn’t exist.

That’s the whole trick of forward-auth. It’s a bouncer standing in front of a door, not a doorman the building hired. The building’s own systems are completely unaware there’s anyone checking IDs outside. This is also exactly why it works for literally any app — Open WebUI, a dashboard, an internal wiki, whatever — regardless of whether that app has ever heard of OIDC, SAML, or any authentication standard at all. If it speaks HTTP, you can put a forward-auth gate in front of it. (This is the exact same forward_auth_block reused later, almost verbatim, to gate a self-hosted SearXNG instance behind Authelia too.)

The tradeoff is just as structural. Because the protected app doesn’t know about Authelia, it also doesn’t really know who’s logged in beyond whatever you choose to pass through those headers. There’s no rich user object, no group claims baked into a token the app can inspect on its own, no session the app controls. Everything the app “knows” about the user is whatever you handed it in a header, and if that header mapping is wrong or missing, the app just falls back to whatever its own default behavior is — which, for Open WebUI, meant new accounts landing in a “pending” status that I had to manually promote the first time around. The app isn’t broken; it’s just genuinely in the dark about the bigger picture.

The app that wants to know you personally

Nextcloud is a different animal entirely, because user_oidc isn’t a gate in front of Nextcloud — it’s a plugin running inside Nextcloud that speaks OpenID Connect directly to Authelia. There’s no bouncer here. Nextcloud itself walks up to Authelia, introduces itself with a client ID and secret, and asks for a proper, structured answer: who is this person, what’s their email, what groups are they in, and here’s a signed token to prove you’re not lying to me.

This is OIDC done the way the spec actually intends it, and it shows in how much more Nextcloud knows once the dance is over. Setting it up means configuring an actual client relationship between the two systems, not just a header-passing convention:

bash

docker exec -u www-data nextcloud php occ user_oidc:provider authelia \
  --clientid="nextcloud" \
  --clientsecret="..." \
  --discoveryuri="https://authelia.yourdomain.com/.well-known/openid-configuration"

That discovery URI is doing a lot of quiet work. Nextcloud fetches it once, learns where Authelia’s authorization endpoint, token endpoint, and signing keys all live, and from then on the two systems are talking a real protocol with real cryptographic guarantees — not just trusting whatever headers happen to show up. When I log in through Nextcloud’s web UI, I’m redirected to Authelia, I authenticate there (TOTP and all), and Authelia hands back a signed JWT containing claims: email, groups, name, sub. Nextcloud reads that token directly and uses it to provision or update my account.

This is the crucial difference. Open WebUI never sees a token — it sees headers Caddy chose to forward. Nextcloud sees an actual cryptographically signed identity document and parses it itself. That gives Nextcloud a much richer, more reliable picture of who’s logging in — group membership for access control, email for notifications, display name for the UI — all sourced directly from the identity provider rather than relayed secondhand through a proxy’s header rewriting.

It also means Nextcloud has opinions about how it builds a user from that token, and those opinions matter more than they look like they should. By default, user_oidc uses the sub claim — a stable, unique identifier Authelia generates — as the internal Nextcloud account ID. That’s the technically correct choice for uniqueness, and it’s also a UUID, not anything resembling a username. I didn’t think about this at all when I set it up, because why would I — the login worked, I could see my files, case closed.

It came back to bite me weeks later in a completely different context (ask me about the desktop sync client some other time), but the shape of the problem is worth flagging here: with forward-auth, the protected app never had to make a decision about identity at all, because it was never given the opportunity. With native OIDC, the protected app is making real decisions — how to map claims to accounts, what becomes the username, whether to provision automatically or require a pre-existing account — and every one of those decisions is a place a default can surprise you.

Same login screen, two different trust relationships

Step back far enough and the distinction is really about where the trust boundary sits. With forward-auth, the trust boundary is at the edge — Caddy and Authelia have a private conversation, and the app behind them is simply handed a verdict. The app trusts Caddy to have done its job; it has no independent way to verify anything, and it doesn’t need one, because it was never designed to.

With native OIDC, the trust boundary moves inside the app itself. Nextcloud doesn’t trust Caddy’s opinion about who’s logged in — Caddy, in fact, is barely involved in the authentication conversation at all beyond passing bytes back and forth. Nextcloud independently verifies a cryptographic token against keys it fetched from Authelia’s discovery document. It’s a direct relationship between the two systems, with the reverse proxy reduced to plumbing.

Neither pattern is strictly better — they solve different problems. Forward-auth is the right tool when you want to blanket-protect something that was never built with SSO in mind, and you’re willing to accept that the protected app will only ever know what you choose to tell it. Native OIDC is the right tool when the app actually needs to reason about identity — assign permissions by group, send mail to the right address, distinguish “logged in as Marco” from “logged in as some anonymous header value” — and you’re willing to take on the deeper integration work and the protocol-level decisions that come with it.

What I’d tell past-me, a few weeks before the desktop sync client incident: when you wire up native OIDC, go look at exactly what claim becomes the account ID, on purpose, before you start using the account for anything real. It’s a five-minute check during setup. It’s a multi-hour debugging session if you skip it and only find out later, indirectly, through a symptom that looks like it’s about something else entirely.

Same login screen. Two completely different conversations happening behind it. Worth knowing which one you’re having before you need to debug it.

This same forward-auth-versus-native-protocol distinction is worth keeping in mind for the network-scoped Authelia bypass too — Open WebUI’s web-search calls to SearXNG hit an API directly, with no forward-auth bouncer able to intercept them the normal way, which is exactly why that post needed a different kind of trust rule instead.

And in many organizations, much of it is theater.

Not because the managers are dishonest. Not because HR has bad intentions. But because the rating that gets written down at the end of the year is shaped by forces that have very little to do with how someone actually performed — and a great deal to do with the company’s financial targets, headcount plans, and budget for the next merit cycle. The result is a system that quietly trades its stated purpose (developing people) for an unstated one (managing cost).

This post is about that gap. Why it exists, how it works, what it costs, and what honest alternatives look like.

It’s an uncomfortable topic. It should be. The people on the receiving end of these ratings deserve a more honest conversation about what they actually represent.

The Stated Purpose vs. the Real One

Officially, the performance review exists to:

• Give people structured feedback on their work.
• Identify development needs and growth opportunities.
• Inform decisions about pay, promotion, and progression.
• Document performance for HR and legal purposes.

That’s the cover story, and most of it is even true — partially. The system does do those things, some of the time.

But sitting underneath all of that is a different reality. In most large organizations, the performance review is also — and often primarily — a budget allocation mechanism dressed up as a meritocracy. The ratings determine the bonus pool distribution. They feed into the merit increase matrix. They justify (or block) promotions, which carry cost implications of their own. They feed into stack rankings, calibration sessions, and — in harder years — into the decisions about who stays and who goes.

The moment a rating has a direct line to a financial outcome, it stops being a pure assessment of performance. It becomes a negotiation between three things: how someone actually performed, what the manager believes they deserve, and what the company can afford to pay out this year.

Guess which one usually wins.

How the Distortion Actually Happens

It doesn’t usually look like fraud. It looks like process. Here are the most common mechanisms by which performance ratings get quietly bent to fit financial targets — patterns documented across years of HR research and reported by managers in companies of every size.

Forced Distributions and “Calibration”

Many organizations operate, formally or informally, on a forced distribution: a set percentage of employees must receive each rating. Maybe 10% top, 70% middle, 20% bottom — or some variation. The stated reason is “to avoid grade inflation.” The actual effect is to cap how many people can be rated as high performers, regardless of how many actually performed at that level.

If your team had a great year and ten people genuinely deserved the top rating, the system doesn’t care. It will let through two or three. The rest will be told they “met expectations” — which, in the language of most companies, quietly means “didn’t quite get there.”

Calibration sessions, where managers gather to “align ratings across the organization,” are where this often happens. The conversation is framed as fairness — and sometimes it is. But it’s also the room where a senior leader can say “we can’t have that many fours this year” and watch ratings get adjusted downward across teams that never compared notes.

The Merit Budget Working Backwards

Here’s the mechanic that’s rarely said out loud but is almost universal: the merit increase budget is set before the ratings are finalized. Finance decides what the company can afford to spend on raises and bonuses for the year. That number gets divided across business units. Each unit gets a budget. And the ratings — which determine who gets what slice of that budget — must add up to something the budget can support.

If finance says the total merit pool is 3%, then the ratings must distribute in a way that the average payout is roughly 3%. If too many people are rated as top performers, the math doesn’t work. So ratings get adjusted until the math does work. The performance assessment, in other words, is constrained by a financial envelope that was decided independently of how anyone actually performed.

This is not a conspiracy. It’s just how most compensation systems are built. But it means the rating you receive is, in part, a function of how good or bad the company’s financial year was — not how good or bad yours was.

The “Save the Rating for Next Year” Move

In a tight year, managers are often told — explicitly or implicitly — to hold back top ratings. The reasoning sounds reasonable: don’t blow the budget now; we’ll recognize them next year. The problem is that next year arrives with a new budget constraint, a new set of priorities, and often a new manager. The rating saved for later rarely materializes.

Worse, the employee never knows. They were told they “met expectations,” and they walk away believing that’s an honest assessment of their year — when in fact their manager spent the calibration session arguing for a higher rating and lost.

Ratings as Layoff Documentation

In organizations preparing for restructuring, performance ratings sometimes serve another function entirely: building a paper trail. If the company anticipates needing to reduce headcount, the ratings issued in the year before can be used to justify who’s let go. This is rarely explicit. But the people who get marked down in a “quiet” year are often the same ones who appear on a list six months later when “performance-based” reductions are announced.

Anyone who has worked in HR or leadership long enough has seen this pattern. It’s one of the most corrosive uses of the performance review system — because it weaponizes a tool that employees were told existed to help them grow.

The Promotion Cap

Promotions almost always carry a cost: higher salary, higher benefits, sometimes higher equity. So in organizations where the promotion budget is constrained, the rating that would justify a promotion — the top tier, usually — becomes scarce by design. Managers are told there’s a “limit” on how many promotions can happen this cycle. That limit translates directly into a limit on how many top ratings can be given out, regardless of who earned them.

People who deserved promotion get told they’re “almost there” or “ready next cycle.” Often, they’re not wrong about being ready. They’re wrong about why the answer was no.

Why Smart Organizations Do This Anyway

It would be easy to frame this as villainous. It isn’t, mostly. The system exists for understandable reasons, and most of the people running it are doing their best inside a structure they didn’t design.

The pressures driving rating distortion are real:

• Cost predictability matters to a business. Investors, boards, and CFOs need to forecast personnel expense. A performance system that produces unconstrained ratings produces unconstrained costs.
• Pure meritocracy is harder than it sounds. If every manager rates their own team honestly, ratings drift upward over time — partly because managers genuinely want to advocate for their people, partly because nobody wants to deliver bad news. Some form of calibration is genuinely necessary.
• Legal and equity considerations require consistency. Wildly inconsistent ratings across managers create legal exposure and real fairness problems, especially when patterns correlate with protected characteristics.
• Limited resources are a real constraint. No company has an infinite bonus pool. Choices have to be made. The question is whether those choices are made honestly or hidden inside the language of performance.

The problem isn’t that financial constraints exist. The problem is that the system pretends they don’t — and asks employees to interpret their rating as a pure assessment of their work when it isn’t.

What This Costs

The cost of dressing up budget decisions as performance assessments is significant, even if it doesn’t show up on any spreadsheet.

Trust erodes. Employees aren’t naive. After a few cycles, most of them figure out that the rating doesn’t quite mean what HR says it means. The slogan becomes the joke. People stop taking the feedback seriously because they can sense it’s been pre-shaped by something other than feedback.

High performers leave. The person who genuinely outperformed and got told they “met expectations” doesn’t argue. They update their resume. The market knows what they’re worth even if their employer pretends not to.

Development conversations get hollow. When the rating is the headline, the conversation around it collapses into justification. The manager spends the meeting defending a number instead of having the harder, more useful conversation about what the person actually did well, where they’re struggling, and what they should work on next.

Managers lose credibility. The people forced to deliver ratings they don’t believe in eventually stop being trusted by their teams. Their feedback becomes suspect. Their authority to coach diminishes. They become message-carriers for a system, not leaders of people.

The wrong behaviors get reinforced. When ratings are tied to visible, measurable outputs (because those are easiest to defend in calibration), the work that can’t be neatly measured — mentoring others, fixing problems before they become visible, building culture, taking on unglamorous work — gets quietly punished. People learn to optimize for what gets rated, not for what matters.

What Honest Alternatives Look Like

There’s no clean, scalable replacement for the performance review. Companies that have tried to abolish them entirely have mostly ended up reinventing them under different names. But there are honest improvements that move the system closer to its stated purpose. Some of them:

Separate the conversations. Pay decisions and development feedback don’t have to live in the same meeting, on the same form, or even in the same quarter. When they’re fused, the development conversation always loses. Splitting them lets each one breathe — and lets feedback be feedback, not the preamble to a number.

Be honest about the budget. Tell people the truth: “The merit pool this year is X percent. Ratings distribute within that envelope. Here’s how I argued for your slice of it.” This is uncomfortable. It’s also vastly more respectful than pretending the budget doesn’t exist.

Drop the forced distribution where possible. Or at least, name it explicitly when it’s in play. The pretense that ratings are independent of distribution caps is one of the most corrosive parts of the system.

Make calibration transparent. Tell people their rating was discussed in calibration, what was raised, and what was concluded. Managers should be willing to say “I argued for a higher rating; here’s why it didn’t go through.” It’s hard. It’s also how trust gets rebuilt.

Decouple performance from pay where you can. Continuous feedback, manager discretion on smaller and more frequent recognition, and project-based bonuses can do more for motivation than the annual rating ever did — without distorting feedback into a budget mechanism.

Train managers to actually give feedback. Most of what passes for performance management is just paperwork because the underlying skill — giving direct, specific, useful feedback in real time — is rare. Investing in that skill does more for actual performance than any ratings system.

Audit for patterns. If certain groups consistently get rated lower, or certain managers consistently rate harder, that’s data worth taking seriously. The honest version of calibration uses this data to surface bias, not to hide it.

None of these are silver bullets. The underlying tension — that companies need to manage cost while also developing people — won’t disappear. But the difference between an honest system and a dishonest one isn’t whether the tension exists. It’s whether the people inside the system are told the truth about it.

What Leaders Can Actually Do

If you’re a manager working inside a system you didn’t design — and most of us are — there are still things within your control. They matter more than the form on the screen.

Tell your people the truth about the system. Don’t pretend the rating is a pure measurement. When you have to deliver a rating that doesn’t reflect what you believe about someone’s performance, say so. Quietly, privately, honestly. “This rating doesn’t capture how I actually see your year. Here’s what I think you did well, and here’s why I couldn’t get the rating I wanted for you.” That conversation costs you nothing and earns enormous trust.

Fight in the room. Calibration is where the real decisions get made. Show up prepared. Make the case. Lose with grace when you have to, but never let a member of your team be downgraded without an argument. If you don’t fight for them in the room, you’re not really their manager — you’re an administrator.

Give feedback all year. The annual review should never be the first time someone hears something important about their work. If it is, that’s a failure of management, not a feature of the system. Build a habit of small, frequent, honest conversations, and the annual rating becomes a footnote.

Distinguish what you control from what you don’t. You probably don’t control the budget. You almost certainly don’t control the rating scale. But you do control how you talk to your people, how you advocate for them, and whether the relationship you build with them is honest. That’s the part that actually matters.

The Bigger Picture

The performance review isn’t broken because the people inside it are bad. It’s broken because it’s been asked to do two incompatible jobs at the same time — develop people and allocate budget — and pretends those jobs are the same job.

They’re not. And the longer organizations pretend otherwise, the more they erode the trust that makes any feedback system work in the first place.

The leaders who navigate this best are the ones who refuse to pretend. They acknowledge the constraints, advocate fiercely for their people, give honest feedback all year long, and treat the annual rating as what it actually is: an imperfect output of a system shaped by forces well beyond performance. They don’t pretend the rating is the truth. They make sure their people hear the truth from them directly.

That’s the part the form can’t do. That’s the part that’s still, and always will be, the job of a human leader.

That world is ending.

Technology and AI are quietly absorbing the parts of a job that used to define competence: pulling reports, drafting emails, summarizing meetings, analyzing spreadsheets, writing first-draft code. The skills that used to make someone a “high performer” — speed, accuracy, technical fluency — are increasingly things a machine does for free, around the clock, without complaint.

What’s left? Exactly the things machines can’t do: imagination, judgment, trust, empathy, the ability to read a room and the ability to change one. These are the deeply human capabilities that no model, however large, can authentically replicate. And unlocking them in other people is what leadership now actually means.

This is what human-centric leadership is about — and it’s not a soft alternative to “real” management. It’s the only kind of leadership that will still matter ten years from now.

The Core Idea

Human-centric leadership puts people at the absolute center of organizational strategy. It treats employees as whole human beings rather than as resources, inputs, or interchangeable units of capacity.

That sounds obvious. It isn’t.

Most organizations still optimize for the machine model under a thin layer of modern language. They publish values about “people first” while running performance systems that reward only what’s measurable. They talk about empathy in town halls while structuring incentives around individual output. They preach psychological safety while penalizing the people who actually speak up.

Human-centric leadership is the discipline of closing that gap. It means designing the real system — the one made of decisions, structures, and daily behaviors — around the assumption that the people in it are full human beings with curiosity, fears, ambitions, off days, and ideas worth hearing. Not in a sentimental way. In a strategic one.

Because here’s the uncomfortable truth: if the analytical and administrative work is being automated, then the only differentiator left is what humans do together. And that depends entirely on whether they’re being led by someone who understands them as humans, or managed by someone who still sees them as headcount.

What’s Actually Shifting

The shift isn’t subtle. A leader’s job used to be roughly:

• Plan the work.
• Assign the work.
• Track the work.
• Report on the work.

Most of that is now either automated or about to be. AI can plan, assign, track, and report — often better than a human manager can, and certainly faster. If that’s all a leader does, the role doesn’t have a long future.

What replaces it is harder, and far more interesting:

• Creating clarity in environments where information moves faster than understanding.
• Building trust in teams that are remote, hybrid, and increasingly cross-cultural.
• Unlocking creativity in work where there’s no playbook, because the playbook was written by the last generation for the last problem.
• Holding meaning when the work is changing so fast that people lose sight of why it matters.
• Coaching judgment — helping people get better at the calls that can’t be reduced to a model.

None of those things show up in a dashboard. All of them determine whether the team succeeds.

The Three Capabilities That Now Define a Leader

If I had to compress human-centric leadership into the three capabilities that matter most right now, they’d be these.

1. Creativity — and creating space for it

AI is excellent at combining what already exists. It’s not yet good at imagining what doesn’t. That gap is where genuine value is created — new products, new approaches, new ways of solving old problems. But creativity is fragile. It dies under constant urgency, micromanagement, and fear of looking foolish.

A human-centric leader’s job is to protect the conditions where creativity is possible: slack in the schedule, permission to explore, tolerance for half-formed ideas, and visible support for the person who proposes something that might not work. That’s not a culture perk. It’s a strategic asset.

2. Collaboration — the actual kind, not the meeting kind

Most organizations have mistaken coordination for collaboration. Coordination is making sure two teams don’t step on each other. Collaboration is two people thinking together and producing something neither could have produced alone. The first is logistics. The second is leadership.

Real collaboration depends on trust, psychological safety, and a shared sense that everyone in the room is treated as a peer regardless of title. None of that emerges by accident. It’s built — slowly, deliberately, and mostly through how the leader behaves when things get hard. Who they listen to. Who they interrupt. Whose ideas they credit. Whose mistakes they cover. These are the small acts that either build collaboration or quietly kill it.

3. Emotional connection — without making it weird

This is the one most leaders shy away from, because the vocabulary is loaded. “Emotional” sounds soft. “Connection” sounds vague. But the underlying capability is concrete: the ability to notice what someone is actually feeling, take it seriously, and respond in a way that respects them as a person.

It shows up in small moments. The leader who notices someone has gone quiet in meetings and asks them about it privately. The one who acknowledges that a reorg is genuinely hard before launching into the rationale for it. The one who says “I don’t know” instead of bluffing. The one who remembers what’s going on in your life when they ask how you’re doing — and waits for the real answer.

People don’t follow leaders who are merely competent. They follow leaders who make them feel seen. In a world where AI handles the competence, the seeing is the job.

What This Looks Like in Practice

Human-centric leadership isn’t a posture or a tone of voice. It’s a set of choices that show up in how you actually run a team. A few that matter most:

Designing work around strengths, not roles. Job descriptions are increasingly fictional. The work that actually needs to be done rarely fits neatly inside a title. Human-centric leaders pay close attention to what each person is genuinely good at and energized by, and shape the work to draw on it — not just because it’s nicer, but because it produces dramatically better results.

Treating one-on-ones as the most important meeting on the calendar. Not status updates. Real conversations. About the work, the person, what’s hard, what’s exciting, what they need from you. If you cancel them when things get busy, you’re sending the message that the relationship is the first thing to go under pressure. That message lands.

Making it safe to disagree with you. If the people around you only ever agree, you’re not leading a team — you’re leading an echo. The signal you send when someone pushes back is one of the most consequential things you do as a leader. Defend the disagreement, even when it stings, and the team will keep telling you the truth.

Being honest about uncertainty. The temptation to pretend you have a plan when you don’t is enormous. Resist it. People can handle “I don’t know yet, here’s how I’m thinking about it” far better than they can handle confident certainty that later turns out to be wrong. The first builds trust. The second burns it.

Caring about the whole person without overreaching. This is the balance. You’re not their therapist, their friend, or their family. But you are a significant presence in their working life, and how you treat them matters. Notice the human. Ask the questions. Respect the boundaries they set. Don’t pry — but don’t pretend the rest of their life doesn’t exist either.

The Honest Counterweight

Human-centric leadership is not soft. It is not the absence of accountability. It is not “being nice” instead of being clear. Some of the most human-centric leaders I’ve worked with are also the most demanding — because they take the people on their team seriously enough to hold them to a high standard, and to tell them the truth when something isn’t working.

The mistake some leaders make when they hear “human-centric” is to drift toward conflict avoidance: softening every message, never giving hard feedback, protecting people from challenge. That isn’t humane. It’s the opposite. It deprives people of the information they need to grow, and it quietly tells them you don’t believe they can handle the truth.

The real version is harder. It’s caring about people enough to be honest with them. It’s holding a high bar because you respect them, not in spite of it. It’s giving feedback that’s direct, specific, and rooted in a genuine belief that they can rise to it.

That kind of leadership is rare. It’s also exactly what the next decade demands.

The Bigger Picture

The shift to human-centric leadership isn’t a trend or a generational preference. It’s a structural response to a structural change. As AI absorbs the analytical and administrative core of knowledge work, the value of leadership migrates upward — away from coordination and toward the deeply human capabilities that determine whether a team can actually do work that matters.

Leaders who get this will build organizations where people do their best work, stay longer, and create things that genuinely move the world forward. Leaders who don’t will spend the next ten years wondering why their best people keep leaving for places that feel more alive.

The choice isn’t really a choice anymore. The machines are getting better at the machine parts of the job. What’s left is the human parts. And those have always been what leadership was actually about — we just had the luxury of pretending otherwise for a while.

That luxury is over. The leaders who recognize it first will define what the next era of work looks like.

Once you have public services running behind OPNsense and HAProxy, you quickly notice something: the internet is noisy. Within hours of exposing port 443, the logs start filling with probes for /wp-login.php, /.env, random PHP shells, and credential stuffing attempts against anything that looks like a login page. Most of it is automated. Almost none of it is your real users.

This post is about the three layers I use to keep that noise out — and to make sure that when something does slip through, it doesn’t get a second try:

1. GeoIP blocking at the firewall, to drop traffic from regions I have no business serving.
2. CrowdSec, to share threat intelligence and block known bad actors automatically.
3. Syncing Fail2Ban logs from my webservers back to OPNsense, so that an attack on a single host turns into a network-wide block.

Each layer is simple on its own. Together, they cut the attack surface dramatically.

The Mental Model

Before the config, the philosophy:

• The firewall should do as much filtering as possible, as early as possible. A packet that’s dropped at OPNsense never reaches HAProxy, never reaches the webserver, and never burns CPU on your application.
• The webserver knows things the firewall doesn’t. It sees failed logins, application-level abuse, and patterns the firewall can’t infer from packet headers alone. That knowledge needs to flow back to the firewall.
• Reputation beats reaction. Blocking an attacker after they’ve hit you is fine. Blocking them before they’ve hit you — because someone else already reported them — is better.

GeoIP handles bulk noise. CrowdSec handles shared reputation. Fail2Ban-to-OPNsense sync handles what only your own services can see. The three together form a feedback loop.

Layer 1: GeoIP Blocking in OPNsense

GeoIP isn’t a security solution by itself — a determined attacker will use a VPN or a cloud instance in a country you allow. But it’s an extraordinarily cheap way to drop the volume of automated attacks, which mostly originate from a predictable handful of regions.

Setup

OPNsense uses MaxMind’s GeoLite2 database via aliases. The high-level steps:

1. Get a free MaxMind license key. Register at maxmind.com and generate a license key under your account.
2. Configure the GeoIP source in OPNsense. Under Firewall → Aliases → GeoIP settings, paste the MaxMind download URL with your license key. OPNsense will pull the database on a schedule.
3. Create GeoIP aliases. Under Firewall → Aliases, create a new alias of type GeoIP, and pick the countries you want to group together. I keep two aliases:
   • GeoIP_Allow — countries where my actual users live.
   • GeoIP_HighRisk — countries that show up disproportionately in my logs as sources of automated attacks.
4. Apply the alias in your WAN-facing rules. On the firewall rule that allows inbound traffic to HAProxy (port 443 from the WireGuard tunnel), add a block rule above it that drops traffic from GeoIP_HighRisk. Or, if you want to be stricter, use a default-deny model: only allow traffic from GeoIP_Allow, and let everything else fall to a deny rule.

A Word of Caution

GeoIP blocking is a blunt tool. Two things to watch for:

• Legitimate users on VPNs or mobile carriers sometimes appear from unexpected countries. If you serve a global audience, default-deny will hurt you. Default-allow with a high-risk blocklist is usually the better compromise.
• CDN and monitoring traffic (uptime checks, search engine crawlers) may originate from regions you’d otherwise block. Whitelist their IP ranges explicitly above the GeoIP rule.

I treat GeoIP as layer one of noise reduction, not as security. It’s the bouncer who turns away the obvious troublemakers before they get to the door.

Layer 2: CrowdSec

CrowdSec is the layer I’m most enthusiastic about. The short pitch: it’s a modern, open-source alternative to Fail2Ban that not only detects local attacks but also shares anonymized signals with a community blocklist. If a thousand other CrowdSec users are being hit by a particular IP, you benefit from that knowledge automatically.

Architecture

There are two pieces:

• The CrowdSec agent — reads logs, detects malicious behavior, and reports it.
• The bouncer — enforces decisions. There’s an official OPNsense plugin that acts as a firewall bouncer, pulling the blocklist and applying it as a pf table.

You can run the agent on OPNsense itself, on your webservers, or both. I run agents on the webservers (where the interesting logs live) and the bouncer on OPNsense (where blocks are most effective).

Setup on OPNsense

1. Install the OPNsense CrowdSec plugin from System → Firmware → Plugins (os-crowdsec).
2. Enroll the instance with the CrowdSec Console (free tier is fine for a homelab). Enrollment lets you manage decisions across multiple agents from one dashboard.
3. Enable the firewall bouncer. It will create a pf table that’s automatically populated with malicious IPs from both your local detections and the community blocklist.
4. Add a block rule referencing the CrowdSec table, placed near the top of your WAN/tunnel-facing rules — above GeoIP and any allow rules.

Setup on the Webservers

1. Install the CrowdSec agent on each webserver (Debian/Ubuntu packages are straightforward).
2. Enable the relevant parsers and scenarios — nginx, apache2, sshd, http-cve, base-http-scenarios, etc. CrowdSec ships with a hub of community-maintained detection rules.
3. Forward decisions to OPNsense. There are two ways:
   • Run a local bouncer on each webserver (e.g., the iptables or nginx bouncer) so blocks happen at the host level too.
   • More importantly, configure the webserver agent to share decisions with the central OPNsense agent via the CrowdSec API. This is where one webserver’s detection becomes a firewall-level block for everything.

Why I Like It

CrowdSec gets you three things Fail2Ban alone doesn’t:

• Community blocklist. You’re blocking known-bad IPs before they ever touch your services.
• Centralized decisions. Multiple agents, one source of truth, one bouncer enforcing at the firewall.
• Better scenarios. Detection rules are written and maintained by a community, not by you at 2 a.m. after an incident.

It doesn’t replace Fail2Ban entirely in my setup — but it carries most of the load.

Layer 3: Syncing Fail2Ban Logs from Webservers to OPNsense

Some things Fail2Ban catches that CrowdSec scenarios might not — custom application logs, niche services, anything you’ve written a bespoke filter for. I still want those bans enforced at the firewall, not just on the host that detected them.

The goal: when Fail2Ban bans an IP on any webserver, that IP gets blocked at OPNsense for everyone.

There are a few ways to do this. Here’s the approach I use, in order of complexity. (Note: this is the API-push method. From App Logs to Firewall Blocks documents a different, pull-based method for the same goal — exporting bans to a flat file that OPNsense fetches on a timer via a URL Table alias, rather than pushing them via API call on every ban. Worth treating as two alternative approaches rather than the same pipeline twice.)

Option A: Fail2Ban → OPNsense Alias via API (Cleanest)

OPNsense exposes a REST API that can manipulate aliases. The flow:

1. Create a host-type alias on OPNsense called something like Fail2Ban_Sync. This alias will hold the IPs to block.
2. Add a firewall rule that drops traffic from Fail2Ban_Sync, placed at the top of your WAN/tunnel rules.
3. On each webserver, configure a Fail2Ban action that, on ban, calls the OPNsense API to add the IP to the alias, and on unban, removes it.

The action script lives in /etc/fail2ban/action.d/opnsense-alias.conf and uses curl to hit the OPNsense API. You’ll need an API key with permission to modify aliases — create a dedicated user with the minimum scope, and store the credentials in a file readable only by root.

A simplified outline of the action:

ini

[Definition]
actionban   = curl -s -u "$API_KEY:$API_SECRET" \
              -X POST "https://opnsense.lan/api/firewall/alias_util/add/Fail2Ban_Sync" \
              -H "Content-Type: application/json" \
              -d '{"address":"<ip>"}'

actionunban = curl -s -u "$API_KEY:$API_SECRET" \
              -X POST "https://opnsense.lan/api/firewall/alias_util/delete/Fail2Ban_Sync" \
              -H "Content-Type: application/json" \
              -d '{"address":"<ip>"}'

Reference that action in your jail.local:

ini

[nginx-http-auth]
enabled  = true
action   = opnsense-alias
filter   = nginx-http-auth
logpath  = /var/log/nginx/error.log
maxretry = 3
bantime  = 86400

Now every ban on the webserver is mirrored to OPNsense within seconds.

Option B: Shipping Logs Centrally and Letting CrowdSec Handle It

If you’re already running CrowdSec, you can also ship webserver logs to a central CrowdSec agent (or use the CrowdSec multi-server setup), and let CrowdSec handle the detection-to-block pipeline. This is cleaner long-term but means moving away from Fail2Ban for those specific filters.

I run a hybrid: CrowdSec for the well-known patterns, Fail2Ban-with-API-sync for the bespoke ones I haven’t ported yet.

Option C: Syslog Forwarding

You can also forward Fail2Ban logs to OPNsense via syslog and parse them there. This is more fragile (parsing logs you didn’t generate is always a tax) and I’d recommend the API approach instead.

Putting It Together: Rule Order Matters

The order of firewall rules on the WAN-facing / tunnel-facing interface matters a lot. Mine looks roughly like this, top to bottom:

1. Allow known-good monitoring and CDN IP ranges (explicit whitelist).
2. Block CrowdSec table.
3. Block Fail2Ban_Sync alias.
4. Block GeoIP_HighRisk alias.
5. Allow HTTPS to HAProxy from the tunnel.
6. Default deny (implicit).

The principle: cheap, high-confidence blocks first; expensive or broader rules later. A packet from a known-bad IP gets dropped before GeoIP even has to look at it.

Operational Notes

A few things I’ve learned running this setup:

• Whitelist yourself. Add your own admin IPs (and your monitoring) to a Whitelist alias that’s referenced above all block rules. The day you lock yourself out because Fail2Ban banned your home IP is the day you build this safeguard — or better, build it before that day, the way Never Lock Yourself Out describes for a dynamic, CGNAT-aware home IP rather than a static one.
• Set sensible bantimes. Permanent bans grow your alias forever and eventually cause performance issues. I use 24 hours for most jails, 7 days for repeat offenders, and a separate manual alias for permanent blocks.
• Monitor the size of your aliases. If your Fail2Ban_Sync alias is growing by thousands of entries a day, something is misconfigured — probably a filter that’s too aggressive.
• Test from outside. Use a VPN or a phone on cellular to verify that blocks actually work from the public side, not just from your LAN.
• Keep an audit trail. Log every API call to OPNsense from your webservers. If you ever wonder why an IP is blocked, you want to be able to answer that question.

The Bigger Picture

What I like about this setup isn’t any one tool — it’s that each layer has a clear job and the layers compose cleanly:

• GeoIP removes bulk noise based on geography.
• CrowdSec removes traffic from IPs that other people have already flagged.
• Fail2Ban sync turns local detections into network-wide blocks.
• HAProxy (from the previous post) decides what gets routed where.
• VLAN segmentation (also from the previous post) ensures that even successful intrusions have a tiny blast radius.

No single layer is doing all the work. If one fails or is misconfigured, the others still hold. That’s the part worth taking away — not the specific commands, but the layered, composable design. It’s the same principle that applies to almost any system worth running: small, focused pieces that each do their job well, and that fail in ways the rest of the system can absorb.

The same composable, defense-in-depth instinct shows up again in Locking Down Authelia — never trusting that one layer rendering correctly (a login page) means the whole chain behind it (2FA, real SMTP delivery, TLS validation) actually works.

If you’re running a similar setup or solving the same problem differently, drop a note. The best part of the homelab community is comparing notes — and the best part of layered security is that everyone’s stack teaches you something new.

But getting traffic into the network is only half the story. Once it arrives, you need a clean, secure, and maintainable way to actually route it to the right service — without accidentally exposing things you didn’t mean to. That’s where the next layer comes in: HAProxy as a reverse proxy, automatic certificates via ACME, and a separate VLAN for services that should never see the public internet.

Here’s how I think about it, and how I built it.

The Problem: One Tunnel, Many Services

The WireGuard tunnel gives me a single entry point. But behind that entry point, I have:

• A handful of services I want to expose (public-facing dashboards, a Git server, a few self-hosted apps).
• A much larger pile of services I never want exposed (internal monitoring, IPMI interfaces, admin panels, backup systems, anything with a weak default).
• TLS certificates I don’t want to manage by hand for every service.

Throwing all of that on one flat network is asking for trouble. So the design splits things up — physically, logically, and at the proxy layer.

The Architecture

At a high level, traffic flows like this:

Internet → VPS (public IP) → WireGuard tunnel → OPNsense
                                                    ↓
                                            HAProxy (DMZ VLAN)
                                            ↙              ↘
                                  Public services       Internal-only services
                                  (DMZ VLAN)            (separate VLAN, no proxy exposure)

Three things matter here:

1. HAProxy lives on a dedicated DMZ VLAN. It’s the only thing that talks to the tunnel-facing side. If it gets compromised, the blast radius is limited to that VLAN.
2. Public services sit on the DMZ VLAN too, but only the ones I’ve explicitly chosen to expose. (This is the same DMZ VLAN that, a few posts later, ends up getting GeoIP blocking, CrowdSec, and a fail2ban sync layered on top of exactly this HAProxy entry point.)
3. Internal-only services live on a separate VLAN entirely. HAProxy can route to them only if I configure it to — and for most of them, I never do. They’re reachable only over the LAN or via WireGuard from my admin devices.

The VLAN separation is enforced at OPNsense with firewall rules, not just at the proxy layer. Defense in depth: even if HAProxy were misconfigured, the firewall wouldn’t let it reach where it shouldn’t.

HAProxy on OPNsense: Why and How

OPNsense ships with an HAProxy plugin, and for a small homelab it’s more than enough. I chose HAProxy over alternatives like Nginx Proxy Manager, Caddy, or Traefik for a few reasons:

• It’s already integrated into OPNsense — no extra container or VM to maintain.
• The configuration model maps cleanly to how I think about traffic: frontends, backends, ACLs, and rules.
• It handles SNI-based routing well, which means I can serve multiple HTTPS services on a single IP and port.

The setup is roughly:

• One frontend on port 443, listening on the DMZ-side interface that receives traffic from the WireGuard tunnel.
• Backends per service, each pointing to the internal IP and port of the actual application.
• ACLs based on SNI / Host header, routing git.example.com to the Git backend, dash.example.com to the dashboard backend, and so on.
• A default backend that returns a generic 404 — anything that doesn’t match an explicit rule gets nothing useful.

That last point matters. Whitelist what you expose. Don’t blacklist what you don’t.

(Caddy ends up playing a closely related role one layer further in, in front of the Docker-based DMZ stack rather than at the OPNsense/HAProxy edge — see Ditch Google for that next hop, where the same “explicit allow, default reject” philosophy shows up again as a Caddy forward_auth block instead of an HAProxy ACL.)

Automatic Certificates with ACME

Managing TLS by hand is a recipe for outages and weekend fire drills. OPNsense has an ACME Client plugin that integrates well with HAProxy.

My setup uses DNS-01 challenges rather than HTTP-01, for two reasons:

1. It works for internal-only hostnames too. I can issue valid Let’s Encrypt certificates for services that aren’t reachable from the public internet, because the validation happens via DNS, not via an inbound HTTP request.
2. It avoids opening port 80 to the outside world for the sole purpose of cert renewal. The WireGuard tunnel + HAProxy already handles 443; I’d rather not introduce another exposed port.

The ACME client talks to my DNS provider’s API, creates the TXT record, gets the cert issued, and hands it off to HAProxy. Renewals happen automatically. I check the logs once a month, and that’s about it.

A small but important detail: I keep a separate account and API token scoped only to the DNS zone used for ACME challenges. If that token leaks, the damage is limited to one zone — not my entire DNS setup. (Worth noting since one TLS mismatch born from exactly this kind of “which name am I actually using” confusion ends up costing a whole debugging session in Locking Down Authelia — a raw IP standing in for a hostname, rather than a DNS-01 zone scoping issue, but the same family of TLS-identity gotcha.)

How I Decide What to Expose

This is the part that’s less about config files and more about discipline. My rule of thumb:

A service gets a public hostname only if it needs one. Everything else stays internal.

Concretely, I ask three questions before exposing anything:

1. Does someone outside my LAN actually need to reach this? If the answer is “it would be convenient,” that’s not a yes. Convenient is what WireGuard is for.
2. Does the service have proper authentication, ideally with MFA? If it’s a service with a single shared password or no auth, it doesn’t get a public route. Period. (This is exactly the bar Authelia, wired up as SSO with real TOTP, ends up clearing for Open WebUI and Nextcloud a few posts later — neither one had MFA on its own.)
3. Am I willing to keep it patched? Exposing something means owning its security posture. If I won’t keep up with updates, it doesn’t go on the proxy.

Anything that fails those checks lives on the internal VLAN and is reachable only via WireGuard from my own devices. That covers the vast majority of my homelab: monitoring, dashboards, IPMI, backup UIs, the lot.

The Mental Model

The way I think about this whole layered setup:

• WireGuard + VPS is the front door. It decides who can knock.
• HAProxy is the receptionist. It decides where the visitor is allowed to go.
• VLANs and firewall rules are the locked doors inside the building. Even if the receptionist makes a mistake, most rooms are still inaccessible.
• ACME keeps the locks well-oiled without me having to think about it.

No single layer is doing all the work. Each one is simple, focused, and replaceable. If I ever swap HAProxy for Caddy or Traefik, the rest of the architecture doesn’t have to change.

That’s the part I find most satisfying — not the specific tools, but the way the layers compose. It’s also, honestly, how I’ve come to think about a lot of things outside the homelab: small, well-defined responsibilities beat one tool trying to do everything.

Everything from here builds on this same foundation: GeoIP/CrowdSec hardening and fail2ban-to-firewall syncing on the WAN side, Authelia SSO and real 2FA on the app side, a self-hosted search engine behind that same Authelia layer, and a dynamic anti-lockout rule for the management plane underneath all of it.