# Filter:drop performance varies by 200x

**URL:** <https://discuss.elastic.co/t/filter-drop-performance-varies-by-200x/115154>\
**Category:** Logstash\
**Created:** [January 11, 2018, 6:32pm UTC](https://discuss.elastic.co/t/filter-drop-performance-varies-by-200x/115154 "2018-01-11T18:32:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Supermathie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/supermathie/32/44936_2.png) [@Supermathie](https://discuss.elastic.co/u/Supermathie)\
**Post date:** [January 11, 2018, 6:32pm UTC](https://discuss.elastic.co/t/filter-drop-performance-varies-by-200x/115154/1 "2018-01-11T18:32:26Z")

</div>

TL;DR: for no apparent reason, logstash will take as low as 0.24ms to drop events, or up to 56ms (a 200x difference!).

I've been trying to track down a performance issue with our logstash setup. We're getting huge backlogs, and it may have to do with the amount of data we're putting into it. We're filtering out a lot of events but still getting poor performance, so last night I started to look at the pipeline performance and see the following for a drop filter:

[https://discourse-cdn-sjc1.com/dev/uploads/default/original/2X/c/ccca6284cbc742d6a179076f6f33580a0d8644a0.png](https://discourse-cdn-sjc1.com/dev/uploads/default/original/2X/c/ccca6284cbc742d6a179076f6f33580a0d8644a0.png)

To play around with the setup and test out locally on my development box, I duplicated a similar pipeline, injecting randomly generated events (some of which get filtered) and got the following:

![image](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b4ce5074ca7e4db93f0447eb735ba54c2214d6eb.png)

Not great performance but similar to what I'm seeing in prod.

After slight changes to the filter (using `else if` instead of `if` one after the other), suddenly I got a drastic performance increase:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/6/0612ec17fe263b54455cb1ca9470a7ce5b65c96a.png)

Then, to verify that this was the cause of the change, I reverted to the original structure, but the performance increase stayed (verification failed):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/7/17dc598b3faa7563a33ee9634c524154cd3b8a0a.png)

At this point I left it for the night.

This morning, I decided to, without restarting anything, re-run the exact same test I did last night. And got a 200x performance difference, with the pipeline viewer reporting it was now taking 50ms/event to DROP messages:

![image](https://us1.discourse-cdn.com/elastic/original/3X/7/1/7147eb3e8d3ed3ea54d0526621446c5ea921f325.png)

This doesn't make any sense to me and makes me feel that something wrong is happening inside of logstash. Help!

* * *

environment:

- logstash 6.0.1
- es 6.0.0
- kibana 6.0.0
- ubuntu 16.04.3
- docker 17.12.0-ce

logstash filter:

```auto
filter {
  mutate {
    convert => [
      "status", "integer",
      "intensity", "integer"
    ]
  }
  # test out the patterns we're using in prod
  if [type] == "rails" {
    if [status] == 200 {
      if [path] =~ /^(\/[^\/]+)?\/magic\/string/ {
        drop { }
      }
      if [path] =~ /^(\/[^\/]+)?\/secret\/sauce/ {
        drop { }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2018, 6:32pm UTC](https://discuss.elastic.co/t/filter-drop-performance-varies-by-200x/115154/2 "2018-02-08T18:32:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.