Request Rate changes at a specific hour

Elastic Stack Elasticsearch
1

Hi i'm handling request rate degradation and I noticed that some of my nodes behaving strange , exactly at 03:04 every night request rate Either drops from 900 to 200 or climb up from 200 to 900

I'm not sure this is normal behavior

image
image846×221 17.5 KB

2

Hi guys ,any Ideas ?

3

We aren't all guys :slight_smile:

What do your Elasticsearch logs for the nodes in question show at this time?

4

:slight_smile: , you are right , sorry
I found the problem , logstash server is bounces with
multiline exception

[2020-08-11T00:58:25,215][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method to_hash_with_metadata' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-multiline-3.0.4/lib/logstash/filters/multiline.rb:273:in block in merge'", "org/jruby/RubyArray.java:1734:in each'", "org/jruby/RubyEnumerable.java:936:in inject'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-multiline-3.0.4/lib/logstash/filters/multiline.rb:272:in merge'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-multiline-3.0.4/lib/logstash/filters/multiline.rb:202:in block in flush'", "org/jruby/RubyHash.java:1343:in each'", "org/jruby/RubyEnumerable.java:830:in map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-multiline-3.0.4/lib/logstash/filters/multiline.rb:200:in flush'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:60:in block in define_flush_method'", "(eval):100:in block in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:602:in block in flush_filters'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:601:in flush_filters'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:642:in flush_filters_to_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:424:in worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384:in `block in start_workers'"]}

the condition in filter looks like :

 if [logger] == "stderr" {
      multiline {
        pattern => "^\t"
        what => previous
      }
    }

please advice what can be done to avoid logstash bounce .

5

It might be better to post something in the Logstash category with a bit more detail.

6

I will , Thank you