Watching the Firewall Breathe: Monitoring OPNsense with Telegraf, InfluxDB, and Grafana

For a long time, “is OPNsense okay?” meant logging into the web UI and eyeballing a handful of widgets on the dashboard. CPU looked fine. Memory looked fine. The interface graphs were… there, technically, but only for whatever window the dashboard widget happened to keep in memory. The moment I closed the tab, that history was gone. If something had spiked at 3am, I had no way to know unless I happened to be awake and staring at the screen when it happened.

That’s not monitoring. That’s just looking.

This post is about the stack I run instead: Telegraf collecting metrics off OPNsense, InfluxDB storing them as a proper time series, and Grafana turning that history into dashboards I can actually trust. None of the three pieces is exotic — this is a genuinely standard combination, sometimes shortened to “TIG stack” minus the L (Telegraf, InfluxDB, Grafana) — but getting OPNsense specifically to feed into it cleanly took a bit more care than I expected, and there are a few decisions along the way that are easy to get wrong without realizing it for weeks.

Why bother with a separate stack at all

OPNsense’s own dashboard isn’t bad. For a quick “is the box currently on fire” check, it’s genuinely useful. But it has three structural limitations that no amount of squinting fixes:

  1. No real history. The graphs reset when you reload the page, and even when they don’t, they’re not backed by a queryable time series — you can’t ask “what did WAN throughput look like at 2am three Tuesdays ago” and get an answer.
  2. No cross-host correlation. If you’re running two OPNsense boxes — which, if you’ve read Never Lock Yourself Out, you know I do, one at home and one fronting the colo stack — the built-in dashboard has no concept of “show me both of these side by side.”
  3. No alerting on the data itself, beyond OPNsense’s own fairly basic notification system. You can get an email when a service crashes, but you can’t easily say “page me if WAN throughput sustains over 80% of link capacity for five minutes” without something purpose-built for that kind of rule.

A proper time series database and a real dashboarding tool solve all three at once, and once it’s running, it becomes the natural home for metrics from everything else in the homelab too — not just OPNsense.

The architecture, in one sentence

OPNsense runs a Telegraf agent that collects local metrics and writes them, over the network, into InfluxDB; Grafana then queries InfluxDB and renders the result as dashboards. Three distinct jobs, three distinct tools, none of them doing more than one thing.

I run InfluxDB and Grafana as Docker containers on the same dmz-stack host that runs Caddy, Authelia, and the rest of the DMZ services I’ve written about before — not on OPNsense itself. OPNsense is a firewall first; I’d rather not load it up with a database and a dashboard server it doesn’t need to be running, and keeping the heavier services off the firewall box means a bad container doesn’t have any chance of taking the firewall down with it.

Step 1 — Installing Telegraf on OPNsense

OPNsense doesn’t ship Telegraf by default, but it’s a one-click plugin install: System → Firmware → Plugins, search for os-telegraf, install it. Once installed, configuration lives under Services → Telegraf, split across a few tabs that map directly onto Telegraf’s own concepts — inputs, outputs, and general settings.

This is worth pausing on, because it’s easy to install the plugin, leave every input checkbox at its default, point the output at InfluxDB, and call it done — and you’ll get something working, but it won’t be the metrics you actually want. Telegraf’s inputs aren’t “on” by default just because the plugin is installed; you choose what gets collected.

The inputs I actually enable:

  • CPU — per-core usage, not just an aggregate average. Aggregate-only CPU graphs hide a single pegged core behind three idle ones.
  • Mem — straightforward, but worth keeping even on a box with plenty of headroom, since a slow memory leak over weeks is exactly the kind of thing a dashboard catches and a quick glance at the live widget never will.
  • Disk / DiskIO — especially relevant if you’re logging anything verbosely to local storage, which on a firewall, you usually are.
  • Interface — per-NIC bytes in/out, errors, drops. This is the single most useful input for a firewall specifically; throughput history per interface is the whole reason most people start this project in the first place.
  • Netstat — connection counts, useful for spotting something hammering the box with new connections faster than normal.
  • Ping — Telegraf can actively ping a target and record latency/loss itself, which is a nice complement to passive interface stats; I point this at my upstream gateway and at the Miami VPS from the CGNAT tunnel post, so a degrading tunnel shows up as a graph trending the wrong way, not just a missed handshake notification.
  • pf states / firewall — OPNsense’s Telegraf plugin exposes pf state table size and related firewall internals, which matters more than it sounds like it should; a state table creeping toward its configured maximum is a real, dashboard-worthy warning sign, not just trivia.

I leave most of the more exotic inputs off. Telegraf’s input list is genuinely huge — it supports things this box will never need — and every enabled input is one more thing writing data, one more thing potentially misbehaving, one more thing to reason about later. Enable what you’ll actually look at.

Step 2 — Pointing Telegraf at InfluxDB

This is the part where version matters more than the OPNsense UI makes obvious. InfluxDB 1.x and InfluxDB 2.x are different enough in their authentication and data model that the Telegraf output configuration looks meaningfully different depending on which one you’re running.

If you’re on InfluxDB 1.x (database/retention-policy model, closer to classic SQL-ish thinking), the output block is the simpler of the two:

toml

If you’re on InfluxDB 2.x (organization/bucket/token model), it’s a different output plugin entirely — influxdb_v2, not influxdb — and authentication is a token, not a username/password pair:

toml

Mixing these up is the single most common way to get a Telegraf service that starts cleanly, logs no errors, and silently writes nothing anywhere — because you’ve configured the v1 plugin against a v2 server (or vice versa), and the connection just fails quietly enough that it’s easy to miss for a while. If your Grafana dashboards are stubbornly empty and Telegraf’s own logs look unremarkable, this mismatch is the first thing I’d check.

I run InfluxDB 2.x, mainly because it’s the actively developed line and the bucket/token model maps more cleanly onto “one bucket per logical source, one scoped token per writer” than the older database/user model did. A scoped write-only token for the OPNsense Telegraf agent, separate from whatever token Grafana uses to read, means a compromised OPNsense box can write garbage into one bucket but can’t touch anything else in the InfluxDB instance.

Step 3 — Wiring Grafana to InfluxDB

Once data is actually landing in InfluxDB — worth confirming with influx query or the InfluxDB UI directly before touching Grafana at all, for the same reason I’d test any pipeline stage in isolation before chaining the next one onto it — adding it as a Grafana data source is the easy part: Connections → Data sources → Add data source → InfluxDB, point it at the same URL, hand it the read token (not the OPNsense write token), and pick the query language.

This is the other place version matters: InfluxDB 1.x speaks InfluxQL, a SQL-like query language; InfluxDB 2.x’s native language is Flux, which is a genuinely different paradigm — pipeline-style, more like a functional data-processing language than a query language. Grafana supports both, but which one you’re writing depends entirely on which InfluxDB version is on the other end, and Flux in particular has a learning curve if you’ve never written it before. I’d budget more time for “learning to write a Flux query that actually does what I want” than for any other single step in this whole setup — it was, for me, the slowest part by a wide margin, slower than anything on the OPNsense or Telegraf side.

A basic Flux query for, say, WAN interface throughput over the last six hours looks like this:

flux

That derivative() call matters more than it looks like it should — Telegraf’s net input reports cumulative byte counters, not instantaneous throughput. Graph the raw counter and you get a relentlessly climbing line that tells you almost nothing useful at a glance; take the derivative and you get the actual bytes-per-second rate, which is the number you actually wanted on the dashboard in the first place. This tripped me up for longer than I’d like to admit — the data was “correct,” it just wasn’t shaped the way a useful graph needs it to be.

Step 4 — Building dashboards that earn their keep

It’s tempting to build one enormous dashboard with every metric on it. Resist that. The dashboards that actually get looked at regularly, in my experience, are the narrow ones:

  • Firewall health — CPU, memory, pf state table usage, disk. The “is this box okay” dashboard, checked reflexively the way you’d check a car’s dashboard lights.
  • Throughput — per-interface bytes in/out, derived to rate, split by interface (WAN, LAN, the WireGuard tunnel interface specifically gets its own panel, since tunnel throughput tells a different story than raw WAN throughput).
  • Tunnel and ping health — the active ping checks against the Miami endpoint and upstream gateway, graphed as latency and packet loss over time, rather than just the binary “handshake or no handshake” signal a status page gives you.

Splitting things this way also makes Grafana’s alerting more useful, since an alert rule tied to a narrow, specific panel (“pf state table over 80% of max for 5 minutes”) is something you can actually reason about and act on, where an alert buried inside a sprawling everything-dashboard tends to get ignored the same way a cluttered notification feed does.

There’s a fourth dashboard worth calling out separately: security signal. Once GeoIP blocking, CrowdSec, and the fail2ban-to-OPNsense sync are all feeding aliases on the firewall, Telegraf’s pf-state and firewall inputs end up indirectly reflecting how much of that traffic is actually getting dropped at the edge — a rising baseline of blocked-packet counts over time is a much better “is the noise increasing” signal than scrolling through raw logs ever was. The same goes for the fail2ban export pipeline — the size of that banlist file is itself a number worth graphing over time, since a sudden jump is exactly the kind of thing a dashboard catches faster than a person remembering to go check.

What I’d tell past-me before starting this

A few things that would have saved real time:

  • Decide your InfluxDB version before writing a single Telegraf config line. Going back and forth between influxdb and influxdb_v2 output plugins after the fact, because you started on one version and migrated, is more annoying than just picking deliberately up front.
  • Enable inputs deliberately, not by leaving every default checked. A Telegraf agent dutifully collecting forty metrics you’ll never look at isn’t free — it’s still I/O, still storage, still cardinality in InfluxDB, for genuinely no benefit.
  • Remember that counters need derivatives. Any “total bytes” or “total packets” style metric is almost always more useful as a rate than as a running total. If a graph looks like a staircase that only ever goes up, that’s usually the tell.
  • Keep write and read tokens separate, the same network-of-least-privilege instinct that shows up everywhere else in this series — scoped API tokens for ACME DNS challenges, scoped tokens for OPNsense’s own REST API, and now scoped tokens for InfluxDB too. It’s the same lesson, applied again.
  • Put Grafana behind Authelia. It’s a web UI with real operational visibility into your network — exactly the kind of thing that deserves the same Authelia SSO and real two-factor setup as anything else in the DMZ, not a standalone admin/admin login sitting on its own.

Where this leaves things

A dashboard that actually has memory now. I can look at three weeks of WAN throughput, correlate a CPU spike with a specific timestamp instead of a vague memory of “it felt slow yesterday,” and get paged by Grafana on a real threshold instead of relying on a script that only checks one binary condition. The OPNsense web UI is still there for day-to-day configuration, but it’s no longer doing double duty as the only window into the firewall’s history — which, in hindsight, was always more than a single page of live widgets was ever going to handle well.

Leave a Comment

Your email address will not be published. Required fields are marked *