# Generate a Detection when new document is indexed
**URL:** https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140
**Category:** Elastic Security
**Tags:** elastic-stack-monitoring, elastic-stack-alerting
**Created:** [March 23, 2021, 9:10pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140 "2021-03-23T21:10:29Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)
#### Post date: [March 23, 2021, 9:10pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/1 "2021-03-23T21:10:29Z")
Hi Community,
I'm trying to generate a detection over time with Elastic SIEM. Giving more context, I'm indexing, through Logstash, data from an API rest, so every half an hour I receive new data. Mostly, is repeated data. So, what I'm trying to achieve is to compare the 30-minute data ago received, with the actual data, and check if there are new documents. Taking this into consideration, the detection should trigger if there are new documents in this time interval in the index.
For example, I received the following documents in an index called `alerts-example` 30 minutes ago:
```auto
{
{
"id":1,
"name": "first alert",
"description": "this is the fist alert"
},
{
"id":2,
"name": "second alert",
"description": "this is the second alert"
}
}
```
And then I receive the new data:
```auto
{
{
"id":1,
"name": "first alert",
"description": "this is the fist alert"
},
{
"id":2,
"name": "second alert",
"description": "this is the second alert"
},
{
"id":3,
"name": "third alert",
"description": "this is the third alert"
}
}
```
so it should detect the last document in the SIEM, in this case:
```auto
{
{
"id":3,
"name": "third alert",
"description": "this is the third alert"
}
}
```
I'm using Elastic 7.12. Is there any possible way or alternative to do this? I would be very grateful for some help.
---
### Author:  [@madi](https://discuss.elastic.co/u/madi)
#### Post date: [March 24, 2021, 8:18pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/2 "2021-03-24T20:18:27Z")
Hi @Felipe_Fuller ... thanks for reaching out!
Am I understanding correctly that you want to receive a) an alert for every new document indexed (N alerts for N documents indexed over the last 30 minute interval)? Or do you want to b) receive only 1 alert every time there are N new documents over the 30 minute interval?
You can accomplish both of these.
For (a), you'd just need to create a "Custom query" rule on `id : *`. If you set the rule interval to 30 minutes, you'll get N alerts every 30 minutes for N new events that are ingested over that interval.
For (b), you can use a "Threshold" rule on `id : *`. If you leave the "Group by" field blank, and set the Threshold to `1`, with a rule interval of 30 minutes, you'll get 1 alert for every 30 minute interval that includes at least 1 new event. You can then use the "Investigate in Timeline" link to get a list of all the original events that are included in the signal.
Hope this helps!
Madi
---
### Author:  [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)
#### Post date: [March 24, 2021, 9:38pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/3 "2021-03-24T21:38:25Z")
Hello @madi ,
Thank you very much for your answer! I am trying to cover the first alternative (a). I will try to implement the first solution you proposed.
To select the 30 minutes period is it necessary to create a new timeline?
As soon as I have a result I will comment it! Again, thank you very much!
---
### Author:  [@madi](https://discuss.elastic.co/u/madi)
#### Post date: [March 29, 2021, 1:58pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/4 "2021-03-29T13:58:28Z")
@Felipe_Fuller It should not be necessary to create a new timeline. You can select the rule interval when creating (or editing) the rule. Happy to provide additional instructions if you are having trouble! Did you have any luck?
Thanks!
Madi
---
### Author:  [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)
#### Post date: [March 29, 2021, 6:21pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/5 "2021-03-29T18:21:01Z")
Hi @madi, thank you very much for all the help. Actually, I tried the method you indicated for point a, and it is reporting all the repeated data. So as an example, if every 30 minutes y receive this data,
```auto
{
{
"id":1,
"name": "first alert",
"description": "this is the fist alert"
},
{
"id":2,
"name": "second alert",
"description": "this is the second alert"
}
}
```
and suddenly, I receive the third one, in this case:
```auto
{
{
"id":1,
"name": "first alert",
"description": "this is the fist alert"
},
{
"id":2,
"name": "second alert",
"description": "this is the second alert"
},
{
"id":3,
"name": "third alert",
"description": "this is the third alert"
}
}
```
I will get the notification for the three documents. Since I was notified before about those other two documents `id: 1 and id: 2`, I don't want to repeat those detections, because each detection is a new issue in Jira.
Thank you in advance!
---
### Author:  [@madi](https://discuss.elastic.co/u/madi)
#### Post date: [March 30, 2021, 2:38am UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/6 "2021-03-30T02:38:04Z")
@Felipe_Fuller Ahh, perhaps I misunderstood. You are receiving the duplicated data every time? In that case, you may be able to de-dupe the data through Logstash (in which case, the rule you set up would work as expected). There are some strategies for de-duping here: [Little Logstash Lessons: Handling Duplicates | Elastic Blog](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)
Let me know if this helps!
Madi
---
### Author:  [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)
#### Post date: [March 30, 2021, 1:11pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/7 "2021-03-30T13:11:04Z")
Hi @madi, thank you so much!! I will take a look at that solution and let you know if that works form me 🙂
Thank you very much!
---
### Author:  [@Felipe\_Fuller](https://discuss.elastic.co/u/Felipe_Fuller)
#### Post date: [April 9, 2021, 6:02pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/8 "2021-04-09T18:02:16Z")
Hi @madi ,
Finally, I redesigned how the data was sent into elastic, so I decided to implement the (a) solution. Thank you very much for your help!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 7, 2021, 6:03pm UTC](https://discuss.elastic.co/t/generate-a-detection-when-new-document-is-indexed/268140/9 "2021-05-07T18:03:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.