# Pause between events ingested in elasticsearch

**URL:** <https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025>\
**Category:** Logstash\
**Created:** [September 9, 2022, 12:33am UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025 "2022-09-09T00:33:57Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 9, 2022, 12:33am UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/1 "2022-09-09T00:33:57Z")

</div>

hello, I would like to know if it is possible, and if there is any filter that can put a time between events before sending them to elastic, for example: I have logs in elastic that are being ingested in the same second, I would like each document had a 30 second delay before being indexed. is there any way to do this?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 9, 2022, 1:36am UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/2 "2022-09-09T01:36:02Z")

</div>

There is nothing built into elasticsearch to delay ingest.... Most folks want the exact opposite ... index events as quick as possible.

You would need to account for this in the client/ log shipping side.

There is a throttle filter in logstash perhaps that will suit your needs

> **[Throttle filter plugin | Logstash Reference \[8.4\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html)**

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 9, 2022, 7:56am UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/3 "2022-09-09T07:56:19Z")

</div>

Try sleep [plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-sleep.html) or you might use ruby code and sleep method to build something custom.

---

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 12, 2022, 6:43pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/4 "2022-09-12T18:43:05Z")

</div>

I tried using the sleep filter, but the documents were still indexed at the same second and thousandth, and I believe this is interfering with my alerts, as I noticed that when multiple logs arrive at the exact same second it doesn't alert all of them, only 1 of them

---

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 12, 2022, 6:45pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/5 "2022-09-12T18:45:45Z")

</div>

hello, could you give me an example of how to use the throatle plugin to do this function? in the case if several logs arrive in exactly the same second, it gives a time between them before sending to elastic, for example, there is a log arriving in the same second and thousandth, I would like each indexed document to have a time of at least 10 seconds before indexing.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 12, 2022, 8:13pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/6 "2022-09-12T20:13:16Z")

</div>

How did you use the `sleep` filter?

Also, did you set `pipeline.workers` to `1` ? Logstash per default will use one worker per CPU core, so if your logstash server has more than one CPU logstash will process multiple events at the same time. To try to achieve what you want you would need to use just one CPU.

Try to set `pipeline.workers` to `1` and use the following `sleep` filter:

```auto
     sleep {
        # Sleep 10 seconds for every event.
        time => "10"
      }

```

This would make Logstash emit an event and sleep for 10 seconds, but this can impact in your ingestion rate.

I'm curious, why you want to do that? If this is impacting in your alerts you need to try to fix on how you are alerting, not change the source events.

---

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 13, 2022, 1:37pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/7 "2022-09-13T13:37:32Z")

</div>

my pipeline workers is set to 3 at the moment, and I put the sleep filter in the same way you mentioned, I'm wanting to do that, because I'm using elastic security's detection and alert function to alert some specific queries in an index , but some logs are arriving at exactly the same second and thousandth and these alerts/logs that arrive at the same time are only being alerted one of them, for example 8 events arrived at the same time but only 1 was issued a notification (in this case I'm using the webhook as a means of notification), I don't know if this is a bug regarding the time of the logs that are arriving at the same time and it's not being able to send via webhook or something I'm doing wrong.

---

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 13, 2022, 1:44pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/8 "2022-09-13T13:44:35Z")

</div>

![Captura de tela de 2022-09-13 10-42-52](https://us1.discourse-cdn.com/elastic/original/3X/b/1/b12df1497c7059260e5a53775d01cd740f972ca9.png)  
here is my pipeline settings at this moment.  
in the pipeline I'm using 2 inputs and 2 outputs for different index

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 13, 2022, 1:49pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/9 "2022-09-13T13:49:40Z")

</div>

Can you share an example of those events and what triggers your alert?

I don't think you will be able to add a delay between your events, and you should not do that, you would be changing the time of the event and this can be misleading in some cases.

---

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 13, 2022, 2:09pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/10 "2022-09-13T14:09:38Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4b6a9598d8d2c504df48946965626b6ea07095c1.png)  
that's what trigger my alert.

and there's a sample off those events

```auto
{
  "_index": "xxx-workloadsecurity-2022.09",
  "_type": "_doc",
  "_id": "vhH4LxxxMBGR_KuvzuGFWN",
  "_version": 1,
  "_score": null,
  "fields": {
    "deviceCustomNumber1.keyword": [
      "266"
    ],
    "deviceEventClassId.keyword": [
      "4000000"
    ],
    "cefVersion.keyword": [
      "0"
    ],
    "severity.keyword": [
      "6"
    ],
    "hostname": [
      "xxxxxxxxxxxxxxx"
    ],
    "TrendMicroDsTenantId.keyword": [
      "xxxx"
    ],
    "syslog_timestamp": [
      "2022-09-12T04:24:04"
    ],
    "deviceVersion.keyword": [
      "xx.x.xxx"
    ],
    "data_e_hora_do_evento": [
      "2022-09-12T01:24:04.000Z"
    ],
    "TrendMicroDsFileSHA1.keyword": [
      "FB0F6C30839XXXXXXXXXXXXXXXXXXXx"
    ],
    "deviceHostName": [
      "example"
    ],
    "deviceAction": [
      "Quarantine"
    ],
    "TrendMicroDsTenantId": [
      "xxxxxx"
    ],
    "result.keyword": [
      "Quarantined"
    ],
    "@version.keyword": [
      "1"
    ],
    "name.keyword": [
      "TROJ_FRS.0NA103BF22"
    ],
    "deviceProduct.keyword": [
      "Deep Security Agent"
    ],
    "deviceEventClassId": [
      "4000000"
    ],
    "tags": [
      "workloadsecurity",
    ],
    "client": [
      "xxxx"
    ],
    "port": [
      xxxx
    ],
    "filePath.keyword": [
      "/local/xxxxxxxx/xxxxxxxxxxxx/cxxxxxxxx.mp4"
    ],
    "message.keyword": [
      "Realtime"
    ],
    "cefVersion": [
      "0"
    ],
    "name": [
      "TROJ_FRS.0NA103BF22"
    ],
    "deviceCustomNumber1": [
      "26"
    ],
    "hostname.keyword": [
      "xxxxxxxxxxxxxxxxxxxxxxxxx"
    ],
    "deviceCustomNumber2": [
      "3404"
    ],
    "cliente.keyword": [
      "xxxxxxxx"
    ],
    "deviceVendor": [
      "Trend Micro"
    ],
    "tags.keyword": [
      "workloadsecurity",
    ],
    "syslog": [
      "471 <134>2022-09-12T04:24:04Z xxxx"
    ],
    "deviceCustomNumber2Label": [
      "Quarantine File Size"
    ],
    "result": [
      "Quarantined"
    ],
    "syslog_timestamp.keyword": [
      "2022-09-12T04:24:04"
    ],
    "deviceVendor.keyword": [
      "Trend Micro"
    ],
    "received_at.keyword": [
      "2022-09-12T04:24:04Z"
    ],
    "@version": [
      "1"
    ],
    "TrendMicroDsTenant.keyword": [
      "599615299724"
    ],
    "deviceProduct": [
      "Deep Security Agent"
    ],
    "deviceCustomNumber1Label": [
      "Host ID"
    ],
    "TrendMicroDsFileSHA1": [
      "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    ],
    "deviceAction.keyword": [
      "Quarantine"
    ],
    "severity": [
      "6"
    ],
    "deviceHostName.keyword": [
      "example"
    ],
    "TrendMicroDsTenant": [
      "xxxxxxxxxx"
    ],
    "filePath": [
      "/local/xxxxxxxxx/xxxxxxxx/xxx.mp4"
    ],
    "deviceCustomNumber2Label.keyword": [
      "Quarantine File Size"
    ],
    "deviceCustomNumber1Label.keyword": [
      "Host ID"
    ],
    "deviceVersion": [
      "50.0.1395"
    ],
    "message": [
      "Realtime"
    ],
    "@timestamp": [
      "2022-09-12T04:32:11.809Z"
    ],
    "syslog.keyword": [
      "471 <134>2022-09-12T04:24:04Z xxxx"
    ],
    "received_at": [
      "2022-09-12T04:24:04Z"
    ],
    "deviceCustomNumber2.keyword": [
      "3404"
    ]
  },
  "highlight": {
    "deviceAction.keyword": [
      "@kibana-highlighted-field@Quarantine@/kibana-highlighted-field@"
    ],
    "deviceAction": [
      "@kibana-highlighted-field@Quarantine@/kibana-highlighted-field@"
    ],
    "deviceCustomNumber2Label": [
      "@kibana-highlighted-field@Quarantine@/kibana-highlighted-field@ File Size"
    ]
  },
  "sort": [
    1662957131809
  ]
}

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 13, 2022, 2:29pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/11 "2022-09-13T14:29:24Z")

</div>

And what is the action of this alert?

Did you set it the Actions Frequency to _On each rule execution_ ?

But I'm not sure if the Detection module will sent an alert for each match, I'm not using it yet because of some limitations on my use case, so I use an external tool.

Another thing is that your filter queries some events that may happen at the same time.

For example, the time logged from your antivirus software Quarantine and Delete could be the same because the precision is only in miliseconds if I'm not wrong.

Maybe you will need to create a rule for each one of those values.

---

<div class="post-metadata">

**Author:** ![RaonyO](https://avatars.discourse-cdn.com/v4/letter/r/5fc32e/32.png) [@RaonyO](https://discuss.elastic.co/u/RaonyO)\
**Post date:** [September 13, 2022, 5:14pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/12 "2022-09-13T17:14:47Z")

</div>

yes it is defined in "each rule detection", I'm thinking that's the problem, it's not alerting all matches.  
About the alert schedule, before it was every 2 minutes, I set it to 0 seconds to see if there was any difference, can you tell me which external tool you use at the moment?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 13, 2022, 5:21pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/13 "2022-09-13T17:21:20Z")

</div>

I'm using [ElastAlert2](https://github.com/jertel/elastalert2) for some alerts, but we are migrating everything to Kibana Alerts and the Detection Rules.

Since there are some limitations with both Kibana Alerts and Detection Rules, some alerts will be triggered by ElastAlert until Elastic improve the detection/alerting system.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 13, 2022, 5:22pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/14 "2022-09-13T17:22:54Z")

</div>

I would suggest that you try to create a different rule for every value you are filtering.

For example a rule only for Delete, another one only for Quarantine etc.

See if it works as you expect.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2022, 5:23pm UTC](https://discuss.elastic.co/t/pause-between-events-ingested-in-elasticsearch/314025/15 "2022-10-11T17:23:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.