Alert Duplicate Check & Deduplication

Alert duplicate check & drop
Some monitoring tools send all active alert events at a given time instead of sending one by one. To integrate with such tools which send a landscape view of alerts, AIOps detects and drops duplicate alerts.

Drop will allow AIOps to drop the duplicate alerts during alert ingestion. If we receive an alert that is an exact duplicate of previous alert, then it will be dropped from further processing

Alert deduplication

Alert deduplication is the process of identifying repeated alert events for the same problem and merging them to a single alert. Merging these alert events into a single alert reduces operational noise by limiting the number of alerts in the system.

This feature in AIOps reduces alert fatigue and enables ITOps team to focus on the important issues.

The following figure depicts various steps in ingesting alert events from monitoring tools and correlating them into meaningful clusters.

Alert Duplicate Check during ingestion

An incoming alert event which is an exact duplicate of an alert event received earlier will be dropped during ingestion and will not be considered for further processing.

Process of Deduplication and Merging

Every alert has its own life cycle. It might start with a warning alert event, progress to a minor warning event, get resolved and then send a clear alert event. Sometimes, it might even flap between these states. AIOps uses deduplication and merging to process alert events to track the life cycle of an alert.

Each incoming alert event has a de-duplication_key field. By default, AIOps autogenerates this key based on the alert source, alert node name and alert metric. This de-dup key defines the context shared by all alerts created due to a same sensor/issue/threshold from a specific alert source. Admins can define the de-dupe key for each alert source.

When AIOps ingests an alert, it compares the new alert with all alerts in non-resolved alert clusters.

If there is a matching alert, AIOps flags the new alert as a duplicate and updates the alert.
If there is no matching alert, AIOps creates a new alert.
In the rare event that there is more than one matching alert, the alert event will be merged with the alert with most recent alert event update.

Alert lifecycle, with event updates are displayed in the Alert Cluster Timeline. Each row in the timeline represents an individual de-duped and merged alert, each dot representing an event in the life cycle of that alert.