Generate a Detection when new document is indexed

Elastic Security
,
1

Hi Community,

I'm trying to generate a detection over time with Elastic SIEM. Giving more context, I'm indexing, through Logstash, data from an API rest, so every half an hour I receive new data. Mostly, is repeated data. So, what I'm trying to achieve is to compare the 30-minute data ago received, with the actual data, and check if there are new documents. Taking this into consideration, the detection should trigger if there are new documents in this time interval in the index.

For example, I received the following documents in an index called alerts-example 30 minutes ago:

{
  {
    "id":1,
    "name": "first alert",
    "description": "this is the fist alert"
  },
  {
    "id":2,
    "name": "second alert",
    "description": "this is the second alert"
  }
}

And then I receive the new data:

{
  {
    "id":1,
    "name": "first alert",
    "description": "this is the fist alert"
  },
  {
    "id":2,
    "name": "second alert",
    "description": "this is the second alert"
  },
  {
    "id":3,
    "name": "third alert",
    "description": "this is the third alert"
  }
}

so it should detect the last document in the SIEM, in this case:

{
  {
    "id":3,
    "name": "third alert",
    "description": "this is the third alert"
  }
}

I'm using Elastic 7.12. Is there any possible way or alternative to do this? I would be very grateful for some help.

2

Hi @Felipe_Fuller ... thanks for reaching out!

Am I understanding correctly that you want to receive a) an alert for every new document indexed (N alerts for N documents indexed over the last 30 minute interval)? Or do you want to b) receive only 1 alert every time there are N new documents over the 30 minute interval?

You can accomplish both of these.

For (a), you'd just need to create a "Custom query" rule on id : *. If you set the rule interval to 30 minutes, you'll get N alerts every 30 minutes for N new events that are ingested over that interval.

For (b), you can use a "Threshold" rule on id : *. If you leave the "Group by" field blank, and set the Threshold to 1, with a rule interval of 30 minutes, you'll get 1 alert for every 30 minute interval that includes at least 1 new event. You can then use the "Investigate in Timeline" link to get a list of all the original events that are included in the signal.

Hope this helps!
Madi

3

Hello @madi ,

Thank you very much for your answer! I am trying to cover the first alternative (a). I will try to implement the first solution you proposed.

To select the 30 minutes period is it necessary to create a new timeline?

As soon as I have a result I will comment it! Again, thank you very much!

1 Like
4

@Felipe_Fuller It should not be necessary to create a new timeline. You can select the rule interval when creating (or editing) the rule. Happy to provide additional instructions if you are having trouble! Did you have any luck?

Thanks!
Madi

1 Like
5

Hi @madi, thank you very much for all the help. Actually, I tried the method you indicated for point a, and it is reporting all the repeated data. So as an example, if every 30 minutes y receive this data,

{
  {
    "id":1,
    "name": "first alert",
    "description": "this is the fist alert"
  },
  {
    "id":2,
    "name": "second alert",
    "description": "this is the second alert"
  }
}

and suddenly, I receive the third one, in this case:

{
  {
    "id":1,
    "name": "first alert",
    "description": "this is the fist alert"
  },
  {
    "id":2,
    "name": "second alert",
    "description": "this is the second alert"
  },
  {
    "id":3,
    "name": "third alert",
    "description": "this is the third alert"
  }
}

I will get the notification for the three documents. Since I was notified before about those other two documents id: 1 and id: 2, I don't want to repeat those detections, because each detection is a new issue in Jira.

Thank you in advance!

6

@Felipe_Fuller Ahh, perhaps I misunderstood. You are receiving the duplicated data every time? In that case, you may be able to de-dupe the data through Logstash (in which case, the rule you set up would work as expected). There are some strategies for de-duping here: Little Logstash Lessons: Handling Duplicates | Elastic Blog

Let me know if this helps!
Madi

1 Like
7

Hi @madi, thank you so much!! I will take a look at that solution and let you know if that works form me :slight_smile:

Thank you very much!

1 Like
8

Hi @madi ,

Finally, I redesigned how the data was sent into elastic, so I decided to implement the (a) solution. Thank you very much for your help!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.