Logstash servers consuming all CPUs

Elastic Stack Logstash
1

Can some advise why logstash using all CPUs
I am using filebeat 5.4 and logstash5.4

image.png690×782 184 KB

logstash config

input {
    beats {
    client_inactivity_timeout => 86400
    port => 5044
    codec => multiline {
      pattern => "^\[%{TIMESTAMP_ISO8601}\]"
      negate => true
      what => previous
     }
    }
}
filter {
#  csv {
#     separator => ":|:"
#     columns => ["Timestamp","Level","hostname","coidkey","Close","Volume", "Volume, "Currency","Weighted", "Price"]
#  }
#}

  mutate {
    gsub => [
      # replace all forward slashes with underscore
      #"fieldname", "/", "_",
      # replace backslashes, question marks, hashes, and minuses
      # with a dot "."
      #"fieldname2", "[\\?#-]", "."
      "message", "\t", " ",
      "message", "\n", " "
    ]
  }
    grok {
    match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp_match}\]%{SPACE}\:\|\:%{SPACE}%{WORD:level}%{SPACE}\:\|\:%{SPA
CE}%{USERNAME:hostname}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:coidkey}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:clientinfo}%{SPACE}\:\
|\:%{SPACE}%{GREEDYDATA:clientip}%{SPACE}\:\|\:%{SPACE}%{GREEDYDATA:Url}%{SPACE}\:\|\:%{SPACE}%{JAVACLASS:class}%{SPACE}\:\
|\:%{SPACE}%{USER:ident}%{SPACE}%{GREEDYDATA:msg}"}
   }
}
output {
    stdout { codec => rubydebug }

  if "_grokparsefailure" in [tags] {
    # write events that didn't match to a file
    file { "path" => "/tmp/grok_failures.txt" }
  } else{
     elasticsearch {
       hosts => "dfsyselastic.df.jabodo.com:9200"
       user => "UN"
       password => "PW"
       index => "vicinio-%{+YYYY.MM.dd}"
       document_type => "log"
     }
   }
}
2

Logstash consuming all CPU isn't necessarily wrong. How many events are you processing every second when you have this load?

Unrelated but mentioning it anyway: Do not use the multiline codec with the beats input. Do the multiline processing on the Filebeat side.

3

I am using 15-20 requests at a time . Not sure how to use multiline on filebeat side , can you please advise on that part .

My log lines are always starting with timestamp but with inconsistent tabs coming in between

[2017-06-26 00:00:00,003]  :|:  INFO   :|:  lvprdsndlbfe1.lv.jabodo.com  :|:    :|:    :|:                                      :|:    :|:  c.m.w.d.s.PixelServiceWrapper                                 :|:   - Start:    Refreshing com.mindspark.webapp.download.service.PixelServiceWrapper

I can not use [ ] to startup point as i have another [ ] coming in between .

4

I am using 15-20 requests at a time

What do you mean? How many events are you processing every second? My point is, if you're processing 10 events per second Logstash shouldn't be using all CPU so if it's doing that there's something wrong, but if it's processing 10000 events per second it's totally normal that it's using CPU.

Not sure how to use multiline on filebeat side , can you please advise on that part .

Have you looked at the Filebeat documentation? I think there's a pretty extensive page on multiline processing. Otherwise ask in the Filebeat group.

5

Thanks I am trying to configure multi line on client side

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.