Monitoring as code: checks that ship with your service

Here's a small opinion I hold strongly: the right time to set up monitoring for a job is the moment you create the job — in the same change, reviewed by the same people. "Later" is where monitoring goes to rot.

I've watched dashboard-configured monitoring drift out of sync with reality more times than I'd like to admit. Someone adds a new nightly job and forgets to add the check. Someone deletes a service but leaves its check behind, cheerfully alerting on a URL nobody pings anymore. Give it six months and your monitoring describes the system as it used to be — and every stale check chips away a little more at your trust in the alerts.

The thing is, this isn't a discipline problem. It's a workflow problem. If remembering to update monitoring is a separate manual step, it will eventually be skipped. So let's not make it a separate step.

Put the check next to the job

When a check is defined in code, right next to the thing it watches, that whole category of drift just… goes away. The check gets created, reviewed, versioned, and deleted with the job. The pull request that adds a backup adds its monitoring. The pull request that removes a service removes the check. Nothing to remember, nothing to leave behind.

That's why I made sure Alwaysbeat could be driven programmatically from day one, not just clicked together in a UI. There are two ways to do it.

The HTTP API

Everything — create, update, pause, delete, list events — is available over a small JSON API with an API key. That's plenty to create checks from a CI step or a provisioning script:

curl -X POST https://api.alwaysbeat.com/api/v1/checks \
  -H "X-DMF-Token: dmf_xxxx" -H "Content-Type: application/json" \
  -d '{
    "name": "nightly-db-backup",
    "schedule": { "kind": "cron", "cron_expr": "0 3 * * *", "tz": "Australia/Melbourne" },
    "grace": "30m",
    "channels": ["email:ops@example.com"]
  }'

The response hands you back the check's ping_url, which you can drop straight into the job you're deploying. The HTTP API reference has the full surface.

Terraform

If you're already describing your infrastructure with Terraform, a check becomes one more resource sitting in the same plan as the service it watches:

resource "alwaysbeat_check" "db_backup" {
  name = "nightly-db-backup"
  schedule {
    kind      = "cron"
    cron_expr = "0 3 * * *"
    tz        = "Australia/Melbourne"
  }
  grace    = "30m"
  channels = ["email:ops@example.com"]
}

Now the monitoring rides along with the same terraform apply that creates the job, and terraform destroy tidies it up afterwards. No orphans, no drift. The Terraform provider docs cover the details and where it's up to.

And close the loop with webhooks

Automation shouldn't stop at creating checks. When a check changes state, Alwaysbeat can POST a signed JSON payload to an endpoint you control — so you can route alerts into whatever you already run, be it a ticketing system, an incident tool, or a chat channel. Each request is signed with HMAC-SHA256 so you can be sure it really came from us; the webhooks reference shows the payload and how to verify it.

Honestly, I think good monitoring should be a little boring: created with the job, torn down with the job, and impossible to forget about. That's the workflow I've tried to build Alwaysbeat around.

— Anton