Could the reason be that previous logs do not have this field?
Their pipelines are exactly the same. Nginx logs were changed (a part was added to them) and thus, I changed grok patterns and added a field to "message". After that, logstash instances kept restarting. logs are shipped to elasticsearch and could be seen on Kibana as well.
Can anyone suggest me a solution to this problem? Thanks in advance!
Hey there, could you please specify the exact version of Logstash and any plugins you use here?
What field exactly is new here?
There might be more information pre and post the error, as well as in the Elasticsearch logs.
Can you please check and see if anything correlates there?
HI Janko! I am using ELK version 7.4.2 (also Filebeat version 7.4.2). The plugins include ip2location and for the filter, I am using Grok. The error logs before the FATAL error:
[2021-07-14T14:35:23,748][ERROR][org.logstash.execution.WorkerLoop][main] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
java.util.ConcurrentModificationException: null
at java.util.IdentityHashMap.forEach(IdentityHashMap.java:1358) ~[?:?]
at org.logstash.Rubyfier.deepMap(Rubyfier.java:71) ~[logstash-core.jar:?]
at org.logstash.Rubyfier.lambda$initConverters$8(Rubyfier.java:99) ~[logstash-core.jar:?]
at org.logstash.Rubyfier.deep(Rubyfier.java:53) ~[logstash-core.jar:?]
at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.ruby_get_field(JrubyEventExtLibrary.java:85) ~[logstash-core.jar:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_ip2location_minus_2_dot_1_dot_0.lib.logstash.filters.ip2location.RUBY$method$filter$0(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ip2location-2.1.0/lib/logstash/filters/ip2location.rb:50) ~[?:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:253) ~[jruby-complete-9.2.8.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.ir.targets.InvokeSite.fail(InvokeSite.java:253) ~[jruby-complete-9.2.8.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162) ~[?:?]
at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.BlockBody.yield(BlockBody.java:114) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.runtime.Block.yield(Block.java:170) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.RubyArray.each(RubyArray.java:1800) ~[jruby-complete-9.2.8.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) ~[jruby-complete-9.2.8.0.jar:?]
at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:99) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:115) ~[logstash-core.jar:?]
at org.logstash.generated.CompiledDataset158.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset159.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset160.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset161.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset162.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset163.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset164.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset165.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset166.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset167.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset168.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset169.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset170.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset171.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset172.compute(Unknown Source) ~[?:?]
at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.8.0.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:243) [jruby-complete-9.2.8.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
If you run with --pipeline.workers 1 does the problem still occur? The Cache.find method is synchronized, so at first glance it should not have concurrency issues, but there may be a corner case...
Hi Badger! Logstash instances are running as a service and I have not changed anything in Logstash configurations for pipeline workers. What does this command do? Doesn't it have an effect on incomming logs?
The workers are set to 2 or 3 on the hosts with Filebeat. Could you please explain more about the problem? I can’t grasp it.
Could it be because previous logs did not include this field?
I had seen this issue initially and now that you also use ip2location, and on the same Elastic stack version, it seems that this might indeed be fitting. It can be hard at times to select the correct parts of a message to search for related issues and, as I mentioned on Slack, more information upfront is always helpful. Had you initially mentioned that this plugin was used and on what version, I would have directly suggested to look in that direction.
Check what it says around the restarting time, maybe some configuration part is missing.
You might need to enable more verbose logging depending on the reason.
That being said, if this didn't help I'd try and reproduce this first on a recent version. The Elastic stack 7.4.2 has been end of life for a couple of months now (Elastic Product End of Life Dates | Elastic) and spending much time digging into issues with old releases can at times be saved this way.
There are also many improvements, performance enhancements and other bug fixes since this version was initially released.
It does not start logstash anymore, also does not write anything (any logs) in /var/log/logstash. It gives me the error while trying it with bin/logstash or the service:
[ERROR] 2021-07-18 10:26:53.732 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) load error: /usr/share/logstash/logstash-core/lib/logstash/runner -- java.lang.RuntimeException: BUG: refined marker called as method
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.