# Elapsed filter doesn't work sometimes

**URL:** https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557
**Category:** Logstash
**Created:** [June 24, 2020, 8:08pm UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557 "2020-06-24T20:08:48Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Ferdous\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_khan/32/49396_2.png) [@Ferdous\_Khan](https://discuss.elastic.co/u/Ferdous_Khan)
#### Post date: [June 24, 2020, 8:08pm UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557/1 "2020-06-24T20:08:48Z")

</div>

Hello,

I am seeing Elapsed filter doesn't work all the times in our staging environment. In our development environment, we have single logstash and elasticsearch instance and Elapsed filter works fine there. But in staging, we have multiple logstash instances. I am not sure whether that's causing the issue or not. So far my observations are:

1. If START and END of a job is within 0-1 second then it fails
2. Sometimes for unknown reason it fails even if the job runs once (no multiple START/END) and finishes within 3600 seconds (Elapsed timeout)

**Example:**

```auto
0787|16132500|Z100_IN_LINE_CUBING_IDOC_WKSUB_2|2020-06-23 16:13:26|START|1|cp_SR0_02|Active
0787|16132500|Z100_IN_LINE_CUBING_IDOC_WKSUB_2|2020-06-23 16:14:26|END|1|cp_SR0_02|Canceled

```

**Here:**  
job\_id 16132500 is used as unique filed  
job\_name is Z100\_IN\_LINE\_CUBING\_IDOC\_WKSUB\_2  
START and END events are tagged as job\_start and job\_finished by Filebeat

**Logstash config:**

```auto
    logstash-host-1$ grep -v \# logstash.yml
    ...
     node.name: abc_indexer_2670
     path.data: .../abc_enablers_indexer
     pipeline.workers: 1
     pipeline.batch.size: 1
     path.config: .../abc_enablers_indexer.conf
    ...
     xpack.monitoring.elasticsearch.hosts: "https://<elk-host-1>:40000"
    ...
     

     
    logstash-host-1$ cat .../abc_enablers_indexer.conf
    input {
            kafka {
                    bootstrap_servers => "<kafka-server-1>:30001,<kafka-server-2>:30001,<kafka-server-3>:30001,<kafka-server-4>:30001"
                    topics => ["abc_enablers"]
                    codec => "json"
            }
    }

    filter {
            date {
                    match => ["ts", "yyyy-MM-dd HH:mm:ss"]
                    target => "@timestamp"
                    locale => "en"
            }
    ...
            elapsed {
            start_tag => "job_start"
            end_tag => "job_end"
            unique_id_field => "job_id"
            periodic_flush => true
            }
    }

    output {
            elasticsearch {
                            hosts => ["https://<elk-host-1>:40000","https://<elk-host-2>:40000","https://<elk-host-3>:40000","https://<elk-host-4>:40000"]
                            index => "abc_enablers"
                            ssl => true
                            ...
    }
  }

```

 ![kibana](https://us1.discourse-cdn.com/elastic/original/3X/2/3/237bebfff350c64544d073efe41e32bb54b3dd5c.jpeg)

Thanks,  
Ferdous

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 24, 2020, 9:37pm UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557/2 "2020-06-24T21:37:38Z")

</div>

The filter is not going to work is the START and END go through different logstash instances. How is traffic allocated to an instance in your staging environment?

---

<div class="post-metadata">

### Author: ![Ferdous\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_khan/32/49396_2.png) [@Ferdous\_Khan](https://discuss.elastic.co/u/Ferdous_Khan)
#### Post date: [June 24, 2020, 10:06pm UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557/3 "2020-06-24T22:06:26Z")

</div>

Thank you for your response. That’s what I thought is causing issue. My guess is it was set as round robin scheme by our ELK admins.

I’m wondering how we could set it such a way that for this particular app (it has separate index and port), traffic will always go to same log stash instance e.g. logstash1. If logstash1 host is down due to maintenance then it will send all traffic to logstash2 host and so son?

Thanks,  
Ferdous

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 24, 2020, 10:44pm UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557/4 "2020-06-24T22:44:34Z")

</div>

Your load balancer admins could configure a VIP using failover rather than round-robin.

---

<div class="post-metadata">

### Author: ![Ferdous\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_khan/32/49396_2.png) [@Ferdous\_Khan](https://discuss.elastic.co/u/Ferdous_Khan)
#### Post date: [June 25, 2020, 12:26am UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557/5 "2020-06-25T00:26:48Z")

</div>

I will check with them. Thank you so much!

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 23, 2020, 12:35am UTC](https://discuss.elastic.co/t/elapsed-filter-doesnt-work-sometimes/238557/6 "2020-07-23T00:35:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.