How can I pattern match a sequence of events?

Elastic Stack Elasticsearch
1

Hi guys, I'am having a steep learning curve with the syntax.
I have a sequence of transactions with timestamps, and when a transaction comes with some values in certain fields, I need to check if in the next 30 seconds some transaction happened with some other values in these fields. This "pair" of transactions with certain pattern is the thing that I need to bucket/flag. I need to make this query over a period of 30 days and send to a dashboard later.

2

Hello Marcos,

What have you already tried? Are you trying this in Kibana or in a Logstash pipeline or an ElasticSearch ingest pipeline?

In LogStash, there is the Aggregate filter for this task.

Best regards
Wolfram

3

This seems to be a perfect use-case for the recently added EQL (Event Query Language).

4

Adding a 3rd option: With transform you can build a secondary index that builds sessions out of your transactions. For your "certain pattern" you can use a scripted metric aggregation. We have an example in the documentation that shows how to calculate durations with transform.

Note that you can write the results to an ingest pipeline after the transform. This gives you even more flexibility. Your need to flag certain records might be easier to do as ingest pipeline, than doing it as part of the transform (+ it's easier to test).

As the result is written to an index, it's easy to build dashboards on top of it.

1 Like
5

@Wolfram_Haussig I can't use Logstash right now, as I received the task specifically to use the ES API with a lambda function (we use dynamodb with streams ON). I need to do this in Kibana. But certainly I will bring the idea of a Logstash pipeline to discuss with my team. Thank you!

@spinscale I will see the EQL, thanks!

@Hendrik_Muhs is x-pack enabled in the basic tier? In the subscriptions page I don't see any information about it. I am really a newbie to this ecossystem. You provided a very nice way to deal with this, I will discuss with my team soon. Thank you!

6

Transform is part of basic, if you scroll down on the subscriptions page you find it listed as "Data Transforms".

That means transform is free to use with the standard distribution of the Elastic Stack.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.