Reading a fleet-wide detection spike

The console shows several hundred detections across the estate, raised within a few minutes of each other, on the same file. The instinct is that something has spread. Usually it has not.

Malware that reaches most of a fleet within minutes is rare and looks specific. A detection engine that changes its mind about a file already present on every machine is common and looks exactly like this. Both are possible. The shape of the spike separates them faster than the file analysis will.

Read the shape first

Timing. Propagation usually leaves a curve: a first host, then a few, then more. An effectively instantaneous spike across the estate points first toward a synchronized event such as a content update, agent update, policy change, or scheduled scan, although it still needs confirmation. Check what changed on the detection platform in the hours before.

Population. Which machines are affected, and which are not? A file present on every machine of one operating system and absent everywhere else is far more likely to be a platform component than an intrusion, which is strong evidence rather than proof. A detection concentrated on a small number of machines that share a function is a much more interesting signal: the engineering team, the finance team, anyone who installed one particular tool. A much smaller number of hosts is more alarming than a larger one here, not less.

Path and provenance. A binary in a system directory, delivered by the operating system vendor, present since the machine was built, with a valid signature, is not what a compromise usually looks like. A binary in a user-writable path, unsigned or recently signed, with a creation time that matches the first detection, is.

Follow-on behavior. Successful malicious execution usually leaves follow-on behavior: network connections, child processes, persistence changes, credential access, or other activity around the process. A false positive often does none of that. If the detection platform shows the flagged process doing nothing except existing, on hundreds of machines, across weeks of history that predate the alert, the file is probably not the event. The detection logic is.

What real compromise tends to look like instead

For contrast, because the same triage instinct has to work in both directions. Developer machines remain productive targets because they tend to combine broad credentials, frequent execution of third-party code and fewer restrictions than many other endpoints. A compromise through developer tooling is therefore more likely to present as a handful of machines, all belonging to engineers, with the modification in project configuration or dependency tooling rather than in a system path, and with network activity to infrastructure nobody recognizes. Small, specific and behavioral. The opposite of a fleet-wide spike on a signed OS binary.

The general point: population size and target coherence carry more information than volume.

The exclusion is the dangerous part

Once the false positive is confirmed, the pressure is to clear the console quickly. This is where lasting damage gets done, because an exclusion is a permanent hole punched in detection and it will outlive everyone’s memory of why it exists.

Some discipline that costs very little:

• Confirm on one host with full telemetry before excluding on all of them. The whole fleet can wait twenty minutes.
• Use the narrowest durable exclusion the platform supports. A cryptographic hash is right for a fixed binary, but it expires with the next version, so for signed software that updates regularly a signer or publisher condition is usually the more durable narrow control. Avoid broad paths, directories and wildcards, and never exclude a folder that users can write to.
• Write down why, in the exclusion itself if the platform has a comment field, and in the change record if it does not.
• Give it a review date. Most exclusions are for a specific version of a specific file and should not survive it.
• Tell the vendor. A false positive that gets corrected upstream removes the need for the exclusion entirely, and vendors generally fix these quickly when given a sample.

Why speed here is a security control

There is a temptation to treat a false-positive storm as an administrative nuisance rather than a security event. It is a security event, because of what it does to the humans.

An estate that generates hundreds of alerts nobody acts on trains its responders to close alerts without reading them. That habit is the actual vulnerability, and it is the mechanism by which the real detection gets acknowledged and dismissed at nine in the morning along with everything else. That detection is the one on four machines, in a developer’s home directory, with outbound traffic.

Tuning is not tidying up. It is preserving the signal.

Explore this expertise: Security Readiness & Response

All technical notes