Creating Multiple Alert Documents when Alert is Triggered

Hi all,

I've got a bit of a unique issue. For the system I am developing, data records will be ingested and compared against thresholds to confirm if values are anomalous.

To test out this functionality I've set up an alert that applies a range query to check if a value is acceptable or not.

I've set up an index connector, so that when the alert is triggered, a document is written to a specific index.

The issue I've encountered is that we require one alert document to match up to each document that has a value out of range (so that when exported to our software we can pair up the alert with the offending document). However, when the alert is triggered, it writes just one alert document with the information about all the offending documents.

The syntax I've used is below, to iterate over the information for each document. But what we want is the ability to write these to separate alert documents. Is this possible?

{
  "flight_id": "[{{#context.alerts}}{{header.fdr_unique_id}}, {{/context.alerts}}]",
  "batch_sequence_counter": "[{{#context.alerts}}{{header.batch_sequence_counter}}, {{/context.alerts}}]",
  "record_sequence_counter": "[{{#context.alerts}}{{header.record_sequence_counter}}, {{/context.alerts}}]",
  "timestamp": "[{{#context.alerts}}{{timestamp}}, {{/context.alerts}}]",
  "rule": "[{{#context.alerts}}{{signal.rule.description}}, {{/context.alerts}}]",
  "navigation_position_estimated_error": "[{{#context.alerts}}{{navigation_position_estimated_error}}, {{/context.alerts}}]"
}

We don't want to use any post-processing if we can avoid it, so any way we can solve this using Elastic functionality would be great.

Hey there @juliette.littlewood, welcome to the community! :wave:

The syntax I've used is below, to iterate over the information for each document. But what we want is the ability to write these to separate alert documents. Is this possible?

So unfortunately, there currently isn't a way to write multiple documents with the Index Connector, you can follow this github issue, but I don't believe there's been much traction there in a bit, so adding a comment would be helpful in prioritization here.


That said, is the intent here to get these documents in a different index than the default .alerts* index that alerts are written to? If not, did the rule you configure at least create individual alert documents for the matching documents as expected?

Just curious to see if there's a way to configure the rule to write the documents as expected without the need of the index action. You might be able to use Building Block Rules to compose them as well. Happy to provide more details if you are willing to share what your base documents and current query looks like.

Cheers!
Garrett

Thank you for your response!
Bit of a shame it can't be done with the Index Connector.

The rule I configured did create individual alert documents for the matching documents, though they are being written into ".internal.alerts-security.alerts-default-*"

Certainly would be happy to try Building Block Rules if that could help. My next idea was to use logstash to transform the alerts, but I wasn't sure if this was the simplest approach.

I'm using a really simple query for the alert at the moment:

{
  "query": {
    "range": {
      "navigation_position_estimated_error": {
        "lt": null,
        "gte": 2
      }
    }
  }
}

The base documents all look similar to this (though some parameters are not available every record):

{
	"header": {
		"fdr_unique_id": "bbf9a814-3d02-4bff-a32e-5ec155ae866d",
		"batch_sequence_counter": "1",
		"record_sequence_counter": "1",
		"record_type": "Update",
		"record_status": "Departed",
		"airline": "BAW",
		"flight_number": "BA0123",
		"origin_airport": "BRS",
		"destination_airport": "LHR",
		"date": "2022-12-13",
		"scheduled_departure_time": "2022-12-13T12:04:38+0000",
		"scheduled_arrival_time": "2022-12-13T15:04:38+0000",
		"actual_departure_time": "2022-12-13T12:34:38+0000",
		"aircraft_registration": "UK-GAANG",
		"icao_aircraft_address": "AC82EC",
		"aircraft_type": "B735"
	},
	"timestamp": "2022-12-13T13:04:39+0000",
	"record_sample_interval": "0.25s",
	"heading": "-92.62572295096697",
	"pitch_attitude": "0.53884008",
	"roll_attitude": "0",
	"navigation_position_lat_long": "[51.46982196273448,-0.4612201950843655]",
	"navigation_position_altitude": "5829.911",
	"navigation_position_ground_speed": "240.093375",
	"navigation_position_track": "-0.12014",
	"navigation_position_estimated_error": "0.8047",
	"normal_acceleration": "0.3735",
	"longitudinal_acceleration": "0",
	"lateral_acceleration": "0",
	"pressure": "199.65733",
	"indicated_airspeed": "240.05436500000002",
	"outside_air_temperature": "3.2497",
	"engine_list": [
		{
			"engine": {
				"engine_number": "1",
				"engine_rpm": "6000"
			}
		},
		{
			"engine": {
				"engine_number": "2",
				"engine_rpm": "5500"
			}
		},
		{
			"engine": {
				"engine_number": "3",
				"engine_rpm": "6500"
			}
		},
		{
			"engine": {
				"engine_number": "4",
				"engine_rpm": "7000"
			}
		}
	],
	"landing_gear_selector_position": "Stowed"
}
1 Like

So yeah, if your goal is to get them to another index your best bet at this point would be to leverage a logstash pipeline like you mentioned (building block rules are all scoped to the same alerts index). Should be able to use the elasticsearch input plugin to fetch the relevant alerts from .internal.alerts-security.alerts-default-*, then transform and output to your desired index.

This should work in the meantime, but please do follow that above github issue for when we're able to enable writing multiple alerts via the index action. I know folks have been working to improve the action experience, so this might be one of the items picked up here soon... :crossed_fingers:

Have a great weekend! :slightly_smiling_face: :wave:

Garrett

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.