Meet Helio: The Real-Time Agent Written in Rust

Every server we run has a small program on it that never sleeps. It's called Helio.

If you run your store on StoreFrame, Helio is the reason your dashboard knows the store took an order before your inbox does. It's also the reason the dashboard can do something about a problem, instead of just drawing a sad red line.

This is the introduction: what Helio is, what it watches and why real-time is the point. After that comes the engineering underneath it, including why your store's history now stays on your own server, and why the agent is now written in Rust.

The short version

Helio is one small container on your server. It does four things:

• It watches the server, the containers your store runs in, the logs, and the store itself.
• It reports all of that, live, to your StoreFrame dashboard over a single connection.
• It listens, so when you press a button in the dashboard, something on your server actually happens.
• It remembers a small history of your store (visitors, shop numbers, security counters) on your own server, not in our database.

That's the whole product. No Magento module, no second agent, nothing installed on your server outside its own container. It runs on every store we operate, and on our own servers too, because I refuse to monitor my infrastructure with something I wouldn't put on yours. It started life in Go. Since November it has been a single Rust binary.

What it watches

Grouped by who tends to care.

The server. CPU, memory, disk and network, every three seconds. How long the store takes to answer. Whether the certificate is healthy, so an expiring one is our problem on Tuesday and not your customers' problem on Sunday.

The containers. Every service your store runs on — web server, PHP, database, cache, search, queue — with its health, restarts and resource use. The container view updates roughly once a second, which is fast enough to watch a reindex eat your memory in real time. Oddly satisfying. Slightly horrifying.

The logs. Container output and the log files that matter, streamed as they're written. Not every five minutes. As they're written.

The store. Order count and revenue, pulled straight from Magento every fifteen seconds. Admin activity. Web traffic — page views, visitors, top pages, referrers, countries — counted from your web server's own access logs, without a single cookie, and with the bots filtered out. And Core Web Vitals from real visitors' browsers, not from a lab test run on a fast laptop.

Security. Who's logged in over SSH, which SSH keys are on the server and when that list changes, and which addresses are being blocked — including why they were blocked.

One connection, both ways

Helio dials out to us. We never dial in. There's no port open on your server for the agent — nothing listening, nothing to knock on. Everything travels over one encrypted WebSocket that Helio opens and keeps open.

Every ten seconds it sends a tiny "still here". Every minute, a full status. So if your server goes quiet, the dashboard knows within about thirty seconds — not when a customer emails.

The same connection carries work back. When you restart a container from the dashboard, look at your scheduled tasks, or check your latest backup, that request travels down to Helio, runs on your server, and the answer comes back up. Every job has a time limit, a size limit on what it returns, and a line in the log saying it happened.

One more thing about logs: the dashboard is the live view, not the archive. Logs are archived somewhere else, on storage your server can't alter. That sounds like a technicality until the day something goes wrong and a server's own logs are gone. It's how we reconstructed an attack to the second in the StyleSmuggler story.

Open the Logs tab and it doesn't start empty. The dashboard asks Helio for the last lines of that log, so you see what just happened before the live stream takes over. It can only ask for logs Helio already streams; it is not a file browser for your server.

Log lines also travel in their own lane. If a container suddenly writes thousands of lines a second, the oldest log lines give way, never the heartbeat or the metrics. A noisy log can't make a healthy server look dead.

Why real-time is the point

Most Magento problems don't announce themselves. A disk fills over a week. A queue backs up over an afternoon. A cache container restarts at 3 a.m. and checkout throws errors for four seconds. By the time a daily report notices, a customer already has.

Seconds-fresh numbers, logs that stream, and a channel that can act on what it sees — that's the difference between finding out and being told. Helio is how every store we run gets the first one.

Your history stays on your server

The obvious way to build a dashboard is to pour everything into one big central database. That's what we did. Every server sent its visitor statistics, its shop numbers and its security counters, and we stored all of it. It works fine with a dozen servers. It is also exactly the kind of thing that gets expensive at a few thousand, because most of what that database wrote all day, more with every server we added, was history that was only ever displayed. None of it was needed to actually run anything.

So Helio keeps it now. Each server has a small SQLite file next to the agent: visitor statistics day by day for 400 days, shop numbers hour by hour for a week, security counters hour by hour for a month. When you open a tab, the dashboard asks that server for the history it needs, and the server answers. Our central database keeps only what we need to manage the servers, such as their settings and when we last heard from them.

It's small. One day of visitor statistics takes a few kilobytes, so two months of our busiest shop comes to about 0.29 MB. And it records whether or not the dashboard is reachable, so an outage on our side no longer leaves a hole in your numbers.

The part I like most isn't the scale, though. Your store's data, and what happens on your server, stays on your server. Anyone working on that server can read the same numbers right there, without asking us. And when a server is retired, its history leaves with it instead of haunting our database.

Why we built our own agent

Before Helio, every server had a small crowd living on it. Not the shop. The things watching the shop. Vector shipped logs. A bash script on a systemd timer collected host metrics. Watchtower pulled new container images. osquery answered questions nobody was asking, and later Netdata drew lovely one-second charts. On top of all that, Ansible over SSH kept asking the most expensive question in operations: is it up?

Let me get the contrarian bit out of the way first: nothing on that list is bad software. Vector is excellent. Watchtower does exactly what it says on the tin. osquery is a genuinely clever idea. Netdata's charts are prettier than anything I will ever build.

The problem was never a single tool. It was the gaps between them.

• Five agents meant five configs, five update cycles and five different opinions about what "healthy" means.
• Stacked together, they were heavy. Roughly 115 MB of RAM per server on paper, and Netdata alone, measured on a busy store, took 314 MB and a steady 12–25% CPU, because it reads everything every second and never idles. That is RAM and CPU the shop never got.
• Each one had its own way of dying quietly. A log shipper that stops shipping looks exactly like a server with nothing to say.
• None of them could talk back. To actually do something about what they saw — restart a container, read a crontab, check a backup — somebody still had to SSH in.
• The data I cared about most wasn't infrastructure data at all. It was orders, revenue, admin logins, the certificate expiring on a Sunday. No generic agent knows what a Magento order is.

At a normal company you shrug and hire a platform team. We are a small team, and every moving part is something one of us personally gets woken up for. Fewer moving parts is not an aesthetic preference. It's sleep.

How one binary is built

Underneath, data goes to up to three places: live over the WebSocket to the dashboard, to the archive through a write-ahead log on disk, and, for the parts worth remembering, into the history on the server itself. If the archive is down, that costs a delay, not data. If the dashboard is down, the history keeps recording.

A few design decisions I'm stubborn about:

• One process. Helio is PID 1 in its container and runs everything as supervised components. No child daemons, no sidecars, no process manager.
• No listening ports. The agent dials out; nothing dials in. There is simply no socket on the server for anyone to knock on.
• It doesn't touch the host. Helio can recreate containers, because containers are disposable. It doesn't install packages, edit host config or tune the kernel. That stays Ansible's job, run centrally and logged.
• Every component looks the same. Setup, Run, Stop, Health. A crashed collector gets restarted with backoff instead of taking the agent down with it.
• Names, not queries. The dashboard can ask a server for a history, or for the last lines of a log, by name. It can't send SQL, a table name or a file path, and Helio only answers for data it already collects.

The whole thing is configured by one YAML file, shaped roughly like this (trimmed and illustrative, not a working config):

agent:
  instance: store-01
  data_dir: /var/helio

sinks:
  otlp:                 # the archive — WAL-backed, survives backend downtime
    type: otlp
    buffer: { type: wal }
  hub:                  # the live view — and the channel jobs come back on
    type: websocket
    reconnect: { jitter: true }

collectors:
  - name: system
    type: host.metrics
    interval: 3s
    sink: [hub]
  - name: containers
    type: docker.logs
    exclude: [helio]    # don't tail your own logs, it gets philosophical
    sink: [otlp, hub]
  - name: ecommerce
    type: exec
    command: scripts/ecommerce.sh
    interval: 15s
    sink: [hub]

history:              # what the server remembers, and for how long
  - name: ecommerce
    source: { type: metrics.exec, name: ecommerce }
    keep: series
    every: 1h
    retention: 168h

The rule for what goes where is boring on purpose: anything fast and numeric is native code inside the agent; anything platform-specific — a database query, a curl, an openssl check — is a small script the agent runs and reads JSON from.

Inspired by the tools that got it right

Very little in Helio is new, and that is on purpose. Most of its best ideas come from tools that solved the same problems first, and solved them well.

• The write-ahead log exists because Promtail taught everyone what happens when a log shipper has no disk buffer.
• The Setup/Run/Stop/Health lifecycle and the per-source rate limiting came from reading Wazuh.
• The heartbeat is kubelet's: a tiny lease every ten seconds, a full status every sixty. The dashboard knows a server has gone quiet within about thirty seconds, without parsing a full payload every ten.
• The container updater is a port of Watchtower's algorithm: compare digests, walk the compose dependencies, stop in reverse order, recreate forward, roll back if the new container won't start.

From Go to Rust, in days

I didn't plan this. Helio was working, and the Go code had barely moved since October: four commits. What changed my mind was Jelle Besseling's Rust work around Magento compilation: Cresset, a set of tools that rebuild Magento's DI compile and static content deploy in Rust, shown during the Mage-OS hackathon in Amsterdam on 4 November. Seeing what Rust could do for Magento's compilation made me ask what it would do for the agent that runs on every server I own.

If you work on Magento, his work is worth an afternoon. magequery answers how a codebase is wired, from DI and plugins to events, cron, routes and config across scopes, without booting the framework. magebuild runs composer install, DI compile, static content deploy and packaging as one parallel build. Small, fast tools that take on the Magento chores everyone grumbles about, and both are open on GitHub.

The case was not speed. Helio runs on every customer server and every host of our own, and it is the one component I can't afford to get wrong. And the failure that worries me most in an agent has one shape: it reports healthy while part of it has quietly stopped.

So the rewrite was parity plus a short, explicit list of fixes, and nothing else. Same config file, same message formats, same log lines my runbooks grep for, so nothing around the agent had to change. I wrote down seventeen specific things the new agent had to do better, then tested it against the Go one, behaviour by behaviour, before it touched a real server.

The spec was written on 6 November. On 9 November every server we run was on the Rust version. That is fast for a rewrite of something this central, and it only worked because the scope was fixed in writing before the first line of code: no new features, every contract pinned by a test.

What came out the other end:

• More than 900 tests, including parity tests against the Go agent's behaviour.
• Unsafe code denied across the whole crate, with exactly one audited exception.
• 57 crates in the whole build, from 11 direct dependencies. Helio's own small HTTP, WebSocket and Docker clients sit on tokio and rustls, instead of a framework for each.
• One build for both x86-64 and arm64 servers.

I wrote it with AI, and that matters for the honest version. Left alone, AI writes lazy Rust: unwrap everywhere, clones to keep the borrow checker quiet. So the brief was strict practice, minimal dependencies and parity tests on every contract. The AI did the typing; the tests did the checking.

What Rust's error handling caught

Be fair to Go first. It served Helio well, and most bugs are logic bugs, which no language prevents. What Rust changed is how every failure reads. A call that can fail returns a Result, and the compiler won't let you ignore it: you decide, right there, what happens next.

Porting line by line under that rule turned up a few places where the Go version had made that decision too lightly, usually by logging an error and carrying on. A job that never returned could hold up every job queued behind it, while health only described the collectors. Job results could be dropped when the outbound buffer was full. Nothing dramatic, but all the same shape: silence that looked like health.

In Rust each became an explicit decision. A job past its timeout is killed together with everything it started. Job results travel in their own queue, ahead of everything else, and survive a dropped connection. Health reports when each part last succeeded, not just that it is running. Log lines later got the same treatment: a queue of their own, so a flood of them can never crowd out a heartbeat. That fits an agent whose worst failure is going quiet.

The other thing Go cost is smaller and constant. A garbage-collected runtime carries its own memory, and an agent on every server pays that on every server. Rust has no garbage collector to pay for.

What changed, measured

Real numbers, read from our servers on 10 and 11 November, so treat them as early. The 42 MB for Go was a busy 8-vCPU store, so the comparison is rough.

The image didn't shrink: it is mostly bash, curl, openssl, jq and sqlite, which both versions carry. The win is memory, roughly a seventh to under half of what the Go version used, and it repeats on every server at once.

Helio next to Grafana Alloy

The obvious question is why not Grafana Alloy. For most people Alloy is the right answer, and I'd say so first.

Alloy wins on breadth, community and the fact that someone else maintains it. Helio exists because we needed four things Alloy isn't built to do: take commands from our dashboard, update containers safely, report store data such as orders and revenue, and keep each store's history on its own server, all through one connection that never listens. I haven't benchmarked Alloy on our servers, so there is no memory number in its column.

Go isn't going anywhere

None of this means we are done with Go. Rust earned Helio because Helio is a long-running agent on every server that has to stay small and never go quiet. Most of what we run is not that, and for a small, focused service Go is still hard to beat: quick to write, quick to compile, one static binary, and a standard library that already speaks HTTP well.

Go still runs at our edge. Every store sits behind OpenResty, NGINX with Lua, and in that same container runs a small Go service that issues and verifies the proof-of-work puzzles behind our self-hosted ALTCHA challenge. It is around 240 lines on top of the official ALTCHA library for Go, and it does one job well. CrowdSec, which decides which addresses get blocked, is written in Go too.

So the choice is not Go versus Rust. It is per job: Rust where a component lives on every server, runs for months and must never go quiet; Go where a small service needs to be simple, fast to build and easy for anyone on the team to read.

Should you do this?

Probably not.

If you run a handful of servers, use the tools. Vector, Prometheus and a proper image-update bot will serve you well, and none of them will ever wake you up over a bug you wrote.

If you have a platform team, same answer, with more dashboards.

It starts to make sense when three things are true at once: you run the same stack on many machines, the data you care about is application data no generic agent understands, and you want the pipe to work both ways — so the dashboard can act, not just watch. That's our situation. It won't be most people's.

And if you build one anyway: keep it boring. One process. Outbound only. A disk buffer. A heartbeat that's cheaper than a status report. Health checks that cover the parts people actually rely on. History kept where the data is born.

If I started again, I'd pick Rust

If I started this today I would build it in Rust, not Go. I wouldn't have said that a year ago. Rust is slower to write by hand. But with AI doing much of the typing and a test suite checking every contract, that cost has nearly gone, while the payoff hasn't: the agent runs on every server, so every megabyte and every memory bug is multiplied by the number of servers.

RAM is not getting cheaper, and an agent at 6–19 MB instead of 42 on every server is memory we get to give back to the shop. And an agent on every server is the last place I want a memory bug. I'd still say what I said above: don't write your own agent unless you have to. But if you do, choose the language that keeps it small and hard to get wrong. Today, for me, that is Rust.