# ConcurrencyError: interrupted waiting for mutex: null in Logstash 2.3.1 **URL:** \ **Category:** Logstash\ **Created:** [April 18, 2016, 8:07am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617 "2016-04-18T08:07:15Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [April 18, 2016, 8:07am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/1 "2016-04-18T08:07:15Z")
Sometimes I have the following issue: > {:timestamp=\>"2016-04-16T04:35:38.758000+0000", :message=\>"Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.", "exception"=\>#\, "backtrace"=\>["org/jruby/ext/thread/Mutex.java:94:in `lock'", "org/jruby/ext/thread/Mutex.java:147:in `synchronize'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:190:in `lazy_initialize'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:268:in `each\_name'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:151:in `each_name'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:150:in `each_name'", "/opt/logstash/vendor/jruby/lib/ruby/1.9/resolv.rb:132:in `getname'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:244:in `getname'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:231:in `retriable\_getname'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:216:in `retriable\_request'", "org/jruby/ext/timeout/Timeout.java:115:in `timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:215:in `retriable\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:230:in `retriable_getname'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:178:in `reverse'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:156:in `reverse'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-dns-2.1.3/lib/logstash/filters/dns.rb:95:in `filter'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:151:in `multi\_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/filters/base.rb:148:in `multi\_filter'", "(eval):10863:in `cond_func_401'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):10851:in `cond_func_401'", "(eval):2035:in `filter\_func'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:267:in `filter_batch'", "org/jruby/RubyArray.java:1613:in `each'", "org/jruby/RubyEnumerable.java:852:in `inject'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:265:in `filter\_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:223:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:201:in `start\_workers'"], :level=\>:error} After that logstash died. ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 7:04am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/2 "2016-08-04T07:04:14Z")
Hi Fedele, sorry for the really late response here. I'm working on a solution for this problem and I'm wondering if you can share a bit more hints about your environment? - OS. - java version. [https://www.elastic.co/guide/en/logstash/current/pipeline.html#\_notes\_on\_pipeline\_configuration\_and\_performance](https://www.elastic.co/guide/en/logstash/current/pipeline.html#_notes_on_pipeline_configuration_and_performance) for more context. Thanks a lot, - purbon ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 4, 2016, 9:10am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/3 "2016-08-04T09:10:41Z")
Hi, don't worry. OS is: ```auto Description: Ubuntu 14.04.5 LTS Release: 14.04 Codename: trusty ``` Java version is: ```auto openjdk version "1.8.0_91" OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14) OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) ``` I tryed some `2.3.x` versions, but I had the same issue. Issue that I don't have with version `logstash 2.2.4`. Maybe it's the new pipeline. Thanks a lot for support ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 10:25am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/4 "2016-08-04T10:25:51Z")
I already identified the situation, is due to a concurrency situation inside the jruby code. I will be releasing a fix in the next days, but in the meanwhile I'm curious for two things: - Do you use config management? if yes, is this config management rewriting somehow the host file? - Do you have an uncommon host file? can you share it might be? Thanks a lot, - purbon ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 4, 2016, 10:54am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/5 "2016-08-04T10:54:45Z")
Sorry, What do you mean with "config management"? Host file == `/etc/hosts`? My ELK configuration is: - 3 nodes (1 client and 2 master) on same ESXi. - On master nodes I have 2 redis servers e 2 logststash indexer I use this configuration to be online/running when I do the updates. All documents go into these redis servers and 2 logstash indexer (with same configuration) take from redis servers and put in elasticsearch with client node. A snippet: ```auto input { redis { host => "server1" data_type => "list" key => "key1" } redis { host => "server2" data_type => "list" key => "key1" } } filter {....} output { if [tag] == "key1" { elasticsearch { hosts => ["client_node"] index => "key1-%{+YYYY.MM.dd}" } } } ``` The issue is on logstash indexer. ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 11:06am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/6 "2016-08-04T11:06:09Z")
Hi, thanks for your quick answer, I mean things like puppet, chef, ansible, etc. The actual source of the problem is related with the `/etc/hosts` file, so I'm wondering if this could be caused by external system like one of this. ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 4, 2016, 11:12am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/7 "2016-08-04T11:12:35Z")
Ok, no `/etc/hosts` does never change. I can confirm this because I use `etckeeper` and this file does not change. ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 11:14am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/8 "2016-08-04T11:14:46Z")
interesting, thanks for this information is very valuable. ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 11:20am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/9 "2016-08-04T11:20:56Z")
Couple of questions out of my head: - Is LS able to read the /etc/hosts file? - Does this file contain some uncommon lines? can you share it might be I would really like to debug more the situation. ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 11:23am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/10 "2016-08-04T11:23:43Z")
I created [https://github.com/logstash-plugins/logstash-filter-dns/issues/26](https://github.com/logstash-plugins/logstash-filter-dns/issues/26) to track the fix. ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 11:48am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/11 "2016-08-04T11:48:14Z")
I created [https://github.com/logstash-plugins/logstash-filter-dns/pull/27](https://github.com/logstash-plugins/logstash-filter-dns/pull/27) as a fix, it would be awesome if you can test the change in your environment and tell me your experience. Thanks for your feedback and help. ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 4, 2016, 2:15pm UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/12 "2016-08-04T14:15:13Z")
LS could read this file, but I think that use DNS because I use dns names in configuration that aren't in `/etc/hosts` My file host is: ```auto 127.0.0.1 localhost 172.x.x.x hostname.mydomain.local hostname # The following lines are desirable for IPv6 capable hosts ::1 localhost ip6-localhost ip6-loopback ff02::1 ip6-allnodes ff02::2 ip6-allrouters ``` ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 4, 2016, 2:25pm UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/13 "2016-08-04T14:25:37Z")
So the issue is in `dns` plugin? In actual configuration I don't have `dns` plugin. But I checked in old version and I used it. If you want I can update logstash for my indexers (that now don't have `dns` plugin) and I can test your change in a shipper (with dns plugin). How can I update plugin with this fix? ---
**Author:** ![purbon](https://avatars.discourse-cdn.com/v4/letter/p/edb3f5/32.png) [@purbon](https://discuss.elastic.co/u/purbon)\ **Post date:** [August 4, 2016, 2:37pm UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/14 "2016-08-04T14:37:53Z")
It would be really nice to validate that the fix is properly working for everyone, having this reproduced in LS alone is being tricky. I will do proper explanation of what you can do tomorrow morning, thanks a lot! ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 4, 2016, 4:44pm UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/15 "2016-08-04T16:44:46Z")
On server with `dns` plugin, logstash reads `/var/cache/nscd/hosts` that is a DNS caching. But with `lsof` I can't see access to `/etc/hosts` ---
**Author:** ![Fedele\_Mantuano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fedele_mantuano/32/9225_2.png) [@Fedele\_Mantuano](https://discuss.elastic.co/u/Fedele_Mantuano)\ **Post date:** [August 17, 2016, 9:04am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/16 "2016-08-17T09:04:54Z")
Hi @purbon. I upgraded to `logstash 2.3.4` where there isn't `dns` plugin, and I didn't have issues. If you have a path, I can test it on my system. I have about 10 milions of logs for day that use this plugin, so if there is a problem I can see it very quickly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:43am UTC](https://discuss.elastic.co/t/concurrencyerror-interrupted-waiting-for-mutex-null-in-logstash-2-3-1/47617/17 "2017-07-06T04:43:05Z")