I have an issue in logstash. It stopped suddenly and no event are parsed.
If i check logstas.err and .stadout no entries apear, BUT in .log it appears an error which is causing this behaviour in logstash.
I copy the entry of my logstash.log
{:timestamp=>"2015-12-02T12:55:31.695000+0100", :message=>"Exception in filterworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", **"exception"=>#<NoMethodError: undefined method []' for nil:NilClass>**, "backtrace"=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-throttle-2.0.2/lib/logstash/filters/throttle.rb:196:infilter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/filters/base.rb:152:in multi_filter'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/filters/base.rb:149:in multi_filter'", "(eval):912:incond_func_13'", "org/jruby/RubyArray.java:1613:in each'", "(eval):896:incond_func_13'", "(eval):486:in filter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:219:infilterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:154:in `start_filters'"], :level=>:error}
As i undesrtand, throttle is causing the pipeline stopping, i marked as bold the part i think is wrong on my logstash and that is causing that, but i don't know what it means and how to fix it.
aditional info: I'm not having a lot of events in my input (around 100 per second).
The crash comes on line 196 when counter(i.e. @event_counters[key]) is nil, but how is that possible given the preceding block that populates @event_counters[key] if it isn't already set?
Unrelated comment: The remove_field option should contain the names of fields, not their contents (which is what you get with %{fieldname}).
Thanks for your reply!!
You are right, it makes no sense that i remove the content of field.
I've already change that and now it seems that is not stopping as often than before.
However, it is stilll stopping.. any other idea?
I have news. AS i change my code, now it doesn't show that error, show another:
:message=>"Exception in filterworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", "exception"=>#<RuntimeError: can't add a new key into hash during iteration>, "backtrace"=>["org/jruby/RubyHash.java:992:in[]='", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-throttle-2.0.2/lib/logstash/filters/throttle.rb:186:in filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/filters/base.rb:152:inmulti_filter'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/filters/base.rb:149:inmulti_filter'", "(eval):1066:in cond_func_18'", "org/jruby/RubyArray.java:1613:ineach'", "(eval):1050:in cond_func_18'", "(eval):531:infilter_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:219:in filterworker'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.0.0-java/lib/logstash/pipeline.rb:154:instart_filters'"], :level=>:error}`
Can i know why is throttle stopping logstash pipeline? it is becomeing a hard problem..
I'm using throttle in order to add a tag when an error appears more than 200 times in 5 minutes, but if there's another way to do that, i use throttle no more.
Marc, what exactly is the key you are looking for to determine a match? Is it simply that the level is 'ERROR'? Also, your config does not jive with what you are saying you need. Try simplifying a bit, maybe? Something similar to:
mutate {
# default a key to something that is probably unique
add_field => { "errorfound" => "${message}" }
}
If [level] == "ERROR" {
mutate {
replace => { "errorfound" => "true" }
}
}
throttle {
key => %{errorfound}
after_count => 200
period => 300
add_tag => "Over200ErrorsIn5Mins"
}
Hi Robert,
The solution you give me doesn't work for my caseYes the key is only to determine if the level i error, BUT, we receive this error from diferent sources and we need to determine the key for each source. That's why we use [%{host}][key]
I've realized that if i check less messages in the if (e.g. [level] == "error" and [host] == "whatever" (or another check)) then it is not stopping the pipeline.
Ok, sorry I'm still mis-understanding something then about what you need the key to be and probably making things worse . It would appear that is what is causing the issues though. So, you can't just use the host as the key when you find that it's an error event like below? Or try %{host}%{mykey} as your key if you have additional data you need to consider along with the host name.
# when an error event arrives, let's keep count per host and tag if we have a large amount
if [level] == "ERROR" {
throttle {
key => %{host}
after_count => 200
period => 300
add_tag => "Over200ErrorsIn5Mins"
}
}
Yes, sorry, throttle should be inisde IF statement.
I use [%{host}][key], to set a variable to then check, but as i see, i can check the host field directly.
Let me check if it works.
Just a question.
If i receive, from several hosts, host field. When throttle check the field it will count even if the content of the field is not the same?
i mean: i want to use throttle when it reaches 100.
-receive in 5 min diferent msg, like 100, from different hosts with host field within.
throttle counts like: 100 totally, or 30 from one server, 40 from another..and so on?
just got back from holiday leave so missed this question earlier. Maybe you can elaborate your specific cases a bit more? It appears you are trying to do too much with one Throttle that is why your questions have varied. What you are saying like needing to count totally for any host and then different counts for individual hosts would require multiple throttles.
I'm trying to explain it in a better way.
I use throttle to check one field. This field exist for certain msg from diferent servers.
I want to throttle for each server isolated (send an e-mail when an especific server reaches 100 msg)
What i'm asking is:
throttle will count it isolated from each server or all together?
Are you running multiple worker threads? I'm getting the same error when using the throttle filter in combination with multiple filter worker threads.
Totday I implemented a throttle filter to dedupe alerts for slack output, and when I restarted logstash I saw a warning "this plugin is not thread safe" , but decided to ignore iT and do some tests. At first everything seemed to run smooth, but a couple of hours later logstash crashed with the same error.
Just as follow up, Im testing the threadsafe version of the trotlle filter now for more then 24 hours without a single crash, while with original throttle filter logstash crashed after less then 5hours
I have configured LS_OPTS="-w 8" in /etc/rc.d/init.d/logstash
The default number of filter workers is 1, but you can increase this number by specifying the -w flag when you run the Logstash agent.
In case you have configured this, the old throttle filter should not be used as its not thread safe, you see a warning about this at startup in the logstash log file.
The link to the modified throttle filter (modifications made by frapex) solves this issue, so you can still use multiple worker threads to improve performance. I'm running with this modified filter since last week, and didn't see any crash anymore.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
. It would appear that is what is causing the issues though. So, you can't just use the host as the key when you find that it's an error event like below? Or try %{host}%{mykey} as your key if you have additional data you need to consider along with the host name.