The machine that is presenting fine in the foreground and dying in the background.

Two principles, worth more than the list

Deltas, not levels

A drive with twenty-four reallocated sectors that has had twenty-four for three years is a scar. The same twenty-four appearing this week is a drive coming apart. Same number, opposite meanings, and only a comparison against an earlier reading tells them apart.

Every counter rule compares the newest check-in against one from seven days ago, and says on screen which of the three cases it is looking at: rising, steady, or not yet knowable. A product that only ever showed you a current number would be handing you the least useful half of the reading.

Unknown is not good news

Where a reading is missing, the hub says so in its own colour rather than leaving a gap. A blank space where a disk health reading should be looks exactly like a healthy disk — and that is the most dangerous thing a monitoring screen can do, because it is indistinguishable from a clean bill of health and it is trusted.

How the figures are gathered

The Linux agent never gains privilege. SMART reads, array state and board sensors need root, so they are taken by a separate root-owned unit that has PrivateNetwork=yes and cannot speak to anything at all. The agent reads the file it leaves behind. There is no sudoers line and no setuid bit. Windows needs no second unit, because the scheduled task already runs as SYSTEM.

Deep readings are taken every thirty minutes rather than every four, because a SMART read wakes a sleeping disk. Doing that three hundred and sixty times a day for years to a drive that is already unwell is not monitoring — it is help.

Disks

The five attributes below are the ones a fleet study of hundreds of thousands of drives found actually correlate with failure. Everything else on a SMART page is interesting; these five are predictive.

Also taken: temperature, power-on hours, power cycles, flash life used, NVMe media errors, unsafe shutdowns, and the NVMe critical-warning bit field — which is decoded into words rather than shown as a number nobody can read.

The raw value, never the normalised one. smartctl reports both: value is the vendor’s opinion rescaled to a hundred-ish, raw is the actual count. Vendors disagree about what a hundred means, and only the raw count can be compared against last week’s.

On Windows these come out of MSStorageDriver_FailurePredictData, a 512-byte blob Windows has always carried and almost nothing reads, decoded by hand. Get-PhysicalDisk and Get-StorageReliabilityCounter supply health status and wear where the machine is new enough to have them.

Mirrors and arrays

The quietest catastrophe there is. A mirror with one leg gone serves every read perfectly and gives no sign at all from the foreground. The machine is fine until the surviving disk goes, and then there is nothing.

• mdadm — array state, level, devices total, active and failed, rebuild progress, and mismatch_cnt. On RAID 1 and 10 a non-zero mismatch count after a check is silent corruption: both copies read back perfectly and one of them is wrong. Nothing else on the machine will ever mention it.
• ZFS — pool health, capacity, fragmentation, and read, write and checksum error counts. A non-zero count on a pool that still says ONLINE is the definition of a machine presenting fine and dying underneath. A pool that has never been scrubbed has never been checked, and that is reported too.
• LVM — only the genuinely silent conditions: a mirrored volume that has lost a leg, and a thin pool filling up. A thin pool that reaches 100% does not return an error politely; it wedges every filesystem sitting on it, at once, without a single disk being full.
• Storage Spaces — virtual disk and pool health. Warning means a column is missing and the data is still there, which is precisely a degraded mirror, so it is never flattened into “not healthy”.

Where a machine has a hardware RAID controller we cannot see behind, the hub says so rather than leaving a blank that reads like a healthy array.

Filesystems

• Read-only remounts. The quietest catastrophe on a Linux server. When ext4 hits an I/O error it remounts read-only to protect what is left. Everything already running carries on from memory, nothing crashes, the site still serves — and every write silently fails until somebody reboots days later and finds the last week does not exist. A machine in this state passes every heartbeat check ever written, including ours.
• Space, and when it runs out. A disk at 82% is not interesting. A disk at 82% that was at 61% last week runs out a week on Thursday, and that is a date somebody can put in a diary.
• Inodes. A filesystem with 400 GB free and no inodes left is full, and every tool that measures free space in bytes says it is fine.
• The NTFS dirty bit, on Windows: set when the filesystem finds something it could not fix in passing, and left set, silently, until somebody reboots into an unexpected chkdsk.

Memory

The quietest failure a server has. A DIMM correcting single-bit errors is doing its job and telling nobody. A module that has thrown one correctable error is far likelier than average to throw an uncorrectable one, and an uncorrectable one is a machine that stops dead in the middle of the night with no explanation attached.

Linux reads the EDAC counters; Windows counts WHEA-Logger events (47 and 19 corrected, 17, 18 and 20 not). Where a physical machine reports memory with no error correction at all, the hub says that plainly — a bit flip there is silent corruption with nothing anywhere to notice it.

Also: swap and page-file use, lifetime page-in and page-out counters, and OOM kills. A machine that kills one process a week for want of memory has a problem it is hiding by murdering something every few days — usually the database, usually at 3am, and usually it restarts before anybody looks.

Pressure

Pressure stall information is the most useful number Linux has added in a decade and almost nothing reads it: the proportion of the last five minutes during which something was waiting for CPU, memory or disk.

Load average cannot tell you this. A load of 8 on an eight-core machine is either perfectly busy or completely stuck, and the number is identical either way. I/O pressure sitting at 40% for an afternoon is a disk that has started retrying reads.

Things that are meant to be running

A failed unit is a job that gave up and told nobody: the backup timer that has been failing since March, the log shipper that died on an upgrade, the fail2ban that has not run since the machine was rebuilt. All of them leave the machine looking perfectly well.

• Linux — systemctl is-system-running, and every failed unit by name. “One unit failed” is not actionable; borgbackup.service failed is.
• Windows — services set to Automatic that are not running, and scheduled tasks whose last run failed, which is where a customer’s own backup script usually lives. Microsoft’s own maintenance tasks and the services Windows deliberately stops are excluded, because a permanent false alarm on every machine is how a monitoring system teaches people to ignore it.

The machine itself

• Thermal throttling. A machine that pulls its own clock down has a blocked fan or a dead one, and it gets slower for months before anybody connects the two.
• Fans reading zero, board temperature, and UPS or battery state.
• Unexpected shutdowns and bugchecks. A machine that has crashed three times this month and come back each time looks perfectly well every time anybody checks on it.
• Windows disk event ids 7, 51, 52, 55, 129 and 153. 129 and 153 — a controller resetting a device that stopped answering, and a retried I/O — catch a disk on its way out months early. They are logged as warnings, nothing surfaces them, and the machine is perfectly usable the whole time. It just occasionally stops for eight seconds and nobody can say why.
• Kernel error counts. A count, never the messages: kernel log lines carry device paths and process names, and that is the customer’s business, not ours.
• A reboot the machine is waiting for. A kernel security update installed six weeks ago on a machine nobody has restarted is not installed — and the update counter reports zero outstanding, which reads as fully patched and is not.
• Restarts, derived. Nothing records a reboot. The hub works it out from uptime going down between two check-ins, which is the only trace an unscheduled restart leaves: the heartbeat resumes and everything looks well.
• The clock, as a fact about the machine. Drift breaks certificate validation, log correlation and domain sign-in, in that order, weeks before anybody notices.
• Network error and drop counters. A count that climbs steadily is a failing cable, a duplex mismatch or a NIC on its way out — and the connection works the whole time. Slowly, with retransmissions nobody sees.

Certificates and backups

Certificates are judged by their issued lifetime, not by days remaining. Caddy’s internal CA issues twelve-hour certificates and renews them constantly; a public-CA thirty-day rule paints such a row amber every single day, and a row that is always amber is a row nobody looks at.

A backup is judged on when it last succeeded, not on what the last run said. A backup that has been failing quietly is only ever discovered on the day it is needed.

What is deliberately not collected

Anything proposed for the payload is judged by one question: if this database leaked, what would the customer mind?

• No file names, user names or process command lines. Counts and byte totals only.
• No kernel or event log messages. Only how many there were.
• No drive serial numbers. They identify a specific asset, they are what a warranty claim is made on, and nothing on the hub needs one. Device names and models are hardware rather than customer data, and are kept because disk health is unreadable without them.
• No network traffic figures. How much data a customer moves is their business; error counters are a health figure, and that is all that travels.
• Nothing inbound. The agent sends. The only thing it reads from the reply is a suggested interval, validated as a bounded integer — see the security position.

What we still cannot see

Said here for the same reason it is said on screen.

• Hardware RAID controllers. PERC, SmartArray and LSI each need their own vendor tool, none of which can be assumed present on a customer machine and none of which this agent will shell out to. Where one is in use the hub reports the blind spot rather than an absent array.
• Disks behind a controller that hides them, and disks on a virtual machine, where the host owns them.
• Anything on a machine without smartmontools installed. The installer says so at install time and the hub says so afterwards.
• Application health. Whether your software is doing its job is a different product. Red Kite watches the machine under it.