Home / Platforms / Operational Alerting

Operational Alerting &
Reporting Automation

Your systems already know when a shovel has been down for forty minutes, when the crusher stopped taking feed, and when a crib break ran twenty minutes long. The information exists and nobody is looking at it, because looking at it is somebody's tenth priority. This watches instead, and puts one message where the people who can act on it already are.

Every operation has already muted an alerting system.

Detection is the easy half. The half that decides whether this is worth anything is whether the channel still gets read in six months, and that is a design problem, not a configuration one. Everything below exists because of it.

Two timelines over the same fault: level-based checking produces twenty-three cards, one per poll for as long as the condition holds, while state-based alerting with a throttle window and a recovery notice produces three
Transitions

Alerts On Change, Not On State

A condition that has been bad for six hours is not six hours of messages. The alert fires when something crosses a line, and then the channel is quiet until something else changes. This single decision is the difference between a tool and a nuisance.

Throttle

One Reminder, Not A Stream

Faults that persist deserve a nudge, not a drip. A configurable window sets how long before a still-broken check speaks again, counted from the original alert rather than reset on every poll, which is the bug that turns hourly reminders into constant ones.

Recovery

It Tells You When It Is Over

A channel that only ever reports failure teaches people that opening it is bad news and they stop. Recoveries are announced, so the channel carries a complete story and reading it is sometimes reassuring.

Honesty

Cannot Tell Is Not The Same As Broken

When a check cannot reach what it monitors, it reports unknown rather than critical. Things that are legitimately absent are marked expected and stay grey. A monitor that cries failure whenever it cannot see is a monitor that gets ignored when it is right.

Dedup

Keyed To The Thing, Not The Poll

Findings are keyed on the business object, a haul cycle, a shift, a machine, so the same event cannot be reported twice no matter how often the service runs or how it is restarted mid-shift.

Reach

No New Application To Adopt

Messages land in a Microsoft Teams or Cisco Webex channel that you populate: dispatch, maintenance planning, the pit supervisor, the superintendent. No app to install, no seat to license, and nothing anyone has to remember to open.

Whatever your shift actually argues about.

These are the ones already running. The list is not a menu so much as an illustration: anything your systems record and nobody watches is a candidate.

Delay And Downtime Thresholds

A machine down longer than the threshold you set, with the delay code, the duration, and who is on it. The point is not the record, it is that somebody hears about it at forty minutes rather than reading about it at end of shift.

While it is still actionable

Meal And Crib Window Compliance

Real-time alerting when a break window runs past its threshold, and an end-of-window summary for supervision. Worth noting what this is for: the loading tool delay Pareto at most operations has meal and crib standby near the top, and it is one of the few categories a supervisor can influence during the shift rather than after it.

Live threshold plus a shift summary

Crusher Feed Interruptions

Feed stopping when it should not be, surfaced immediately rather than discovered in the morning reconciliation of why the mill was short.

Minutes, not the next morning

Pace Against Plan

Projected end of shift falling below plan by a margin you set, early enough that moving a truck or opening a second face is still a decision rather than a post mortem.

Intervene, not report

Material Misroutes

Wrong-destination dumps caught against the block model and the destination configuration, with four independent detection layers.

Has its own platform
Explore misroute detection →

Shift And Daily Report Automation

The end-of-shift summary, the daily pack, and the workbook somebody currently assembles by hand, generated and delivered on a schedule with nobody in the loop.

The five a.m. spreadsheet, retired

Some problems have no number to set.

Everything above fires when something crosses a line somebody chose. That covers a great deal, and it cannot cover drift: a haul route that has quietly become five percent slower has no threshold, because nobody knows what that route's normal is and its normal changes every time the pit does. The Operations Advisor works the other way round. It learns what normal was, per route, per machine, per face, and reports the departures with the effect size, the sample it is standing on, and a confidence figure.

It is built on the same delivery layer as everything on this page, so the findings arrive in the same channel, graded so they can be triaged by scrolling. It is scoped as its own engagement because it needs something the others do not: a period of accumulated history at your site before it is entitled to an opinion.

Something has to watch the watchers.

An alerting service that dies quietly is worse than no alerting service, because the silence reads as good news. So the stack monitors itself on one page: services, scheduled jobs, ports and HTTP responses, containers, CPU and memory, disk headroom, GPU load, and the freshness of every dataset the reports depend on. Forty-three checks on the portal below, each one clickable through to what it means, how to restart it, and what to look at if restarting does not fix it.

That last part is the difference between documentation and a runbook. The guidance lives on the check itself, so the person looking at a red card at two in the morning is one click from the fix rather than searching a shared drive for a procedure written by somebody who has since left.

The platform health portal: an overall status of OK across forty-three checks, grouped into services, timers, ports and HTTP, containers, system CPU and memory, disks, GPU, and data freshness, each rendered as a card with its current state and detail
Alerts arriving in a Teams channel: misroute detections with truck, operator, payload and destination, alongside an end-of-shift meal delay compliance summary
Source excerpt from the health watchdog showing how a scheduled job is checked: state alone is insufficient because a timer can report itself active while having stopped firing, so the last trigger time is compared against a maximum age

The failure mode that excerpt exists for: a scheduled job can report itself perfectly healthy while having quietly stopped running. Checking whether it is enabled tells you nothing. Comparing its last run against a maximum age is what catches a collector that stopped feeding your reports three days ago, which is the kind of thing normally discovered by someone asking why a number looks wrong.

Screenshots from production systems. Site and personnel identifiers have been altered. Line numbers are genuine; schema identifiers elsewhere on this site are masked.

What this replaces.

Finding Out At Handover

Most operational problems are already recorded somewhere by the time anyone acts on them. The cost is not the missing data, it is the hours between the event and somebody noticing, and those hours are where production goes.

Same event, hours earlier

The Premium Connector Tax

Cloud automation platforms bundle a free tier for mail and spreadsheets and classify your site database as premium, so every automation worth building falls outside it. From there it is per-user licences, per-flow subscriptions, or metered per-run billing, and a check that runs every fifteen minutes runs ninety-six times a day. A Python service under systemd on your own server has no meter on it at all.

No licence, no capacity tier, no per-run charge

Automation That Waits On Approval

The more common cost is not a bill, it is finished work sitting undeployed while a licence request moves through procurement. Nothing on your own infrastructure needs a purchase order before it can start running.

Built Friday, running Monday

Two to three weeks per alert family.

What did you find out too late this month?

That question is usually the whole scoping conversation. Tell us the event, and whether your systems already record it decides how quickly it can be watched.