Filter:drop performance varies by 200x


(Michael Brown) #1

TL;DR: for no apparent reason, logstash will take as low as 0.24ms to drop events, or up to 56ms (a 200x difference!).

I've been trying to track down a performance issue with our logstash setup. We're getting huge backlogs, and it may have to do with the amount of data we're putting into it. We're filtering out a lot of events but still getting poor performance, so last night I started to look at the pipeline performance and see the following for a drop filter:

To play around with the setup and test out locally on my development box, I duplicated a similar pipeline, injecting randomly generated events (some of which get filtered) and got the following:

image

Not great performance but similar to what I'm seeing in prod.

After slight changes to the filter (using else if instead of if one after the other), suddenly I got a drastic performance increase:

Then, to verify that this was the cause of the change, I reverted to the original structure, but the performance increase stayed (verification failed):

At this point I left it for the night.

This morning, I decided to, without restarting anything, re-run the exact same test I did last night. And got a 200x performance difference, with the pipeline viewer reporting it was now taking 50ms/event to DROP messages:

This doesn't make any sense to me and makes me feel that something wrong is happening inside of logstash. Help!


environment:

  • logstash 6.0.1
  • es 6.0.0
  • kibana 6.0.0
  • ubuntu 16.04.3
  • docker 17.12.0-ce

logstash filter:

filter {
  mutate {
    convert => [
      "status", "integer",
      "intensity", "integer"
    ]
  }
  # test out the patterns we're using in prod
  if [type] == "rails" {
    if [status] == 200 {
      if [path] =~ /^(\/[^\/]+)?\/magic\/string/ {
        drop { }
      }
      if [path] =~ /^(\/[^\/]+)?\/secret\/sauce/ {
        drop { }
      }
    }
  }
}

(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.