# Elasticsearch blocked futex **URL:** \ **Category:** Elasticsearch\ **Created:** [September 24, 2014, 10:16pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953 "2014-09-24T22:16:51Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 24, 2014, 10:16pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/1 "2014-09-24T22:16:51Z")
If anyone can help me understand why my cluster is hung I would appreciate it. jstack output: > I am able to query the cluster and health is good but I can't DELETE or CLOSE index as it is unresponsive. mlockall is set to true iostat: avg-cpu: %user %nice %system %iowait %steal %idle 2.00 0.05 0.30 0.08 0.00 97.57 Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sdb 7.40 0.00 939.20 0 4696 sda 0.40 0.00 4.80 0 24 dm-0 0.60 0.00 4.80 0 24 dm-1 0.00 0.00 0.00 0 0 dm-2 117.40 0.00 939.20 0 4696 avg-cpu: %user %nice %system %iowait %steal %idle 2.93 0.03 0.23 0.08 0.00 96.74 Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sdb 6.80 0.00 776.00 0 3880 sda 0.80 0.00 20.80 0 104 dm-0 2.60 0.00 20.80 0 104 dm-1 0.00 0.00 0.00 0 0 dm-2 97.00 0.00 776.00 0 3880 avg-cpu: %user %nice %system %iowait %steal %idle 1.20 0.03 0.25 0.10 0.00 98.42 Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sdb 11.40 0.00 1312.00 0 6560 sda 0.80 0.00 22.40 0 112 dm-0 2.80 0.00 22.40 0 112 dm-1 0.00 0.00 0.00 0 0 dm-2 164.00 0.00 1312.00 0 6560 avg-cpu: %user %nice %system %iowait %steal %idle 7.07 0.03 0.50 0.08 0.00 92.33 Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sdb 20.40 0.00 5064.00 0 25320 sda 1.00 0.00 25.60 0 128 dm-0 3.20 0.00 25.60 0 128 dm-1 0.00 0.00 0.00 0 0 dm-2 633.00 0.00 5064.00 0 25320 avg-cpu: %user %nice %system %iowait %steal %idle 1.23 0.05 0.33 0.10 0.00 98.30 Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn sdb 15.20 0.00 2604.80 0 13024 sda 2.40 0.00 38.40 0 192 dm-0 4.80 0.00 38.40 0 192 dm-1 0.00 0.00 0.00 0 0 dm-2 325.60 0.00 2604.80 0 13024 vmstat: -bash-4.1$ vmstat 5 procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 96 2 0 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 97 0 0 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 98 0 0 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 95 0 0 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 95 0 0 -bash-4.1$ lsof -u elasticsearch | wc -l 3004 /etc/security/limits.conf:elasticsearch hard nofile 65536 /etc/security/limits.conf:elasticsearch soft nofile 65536 /etc/security/limits.conf:elasticsearch - memlock unlimited top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers Swap: 0k total, 0k used, 0k free, 1963024k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 24, 2014, 10:20pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/2 "2014-09-24T22:20:15Z")
What state is your cluster in? Can you get \_cat/health to return? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 25 September 2014 08:16, Chris Denneen [cdenneen@gmail.com](mailto:cdenneen@gmail.com) wrote: > If anyone can help me understand why my cluster is hung I would appreciate > it. > > jstack output: > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > I am able to query the cluster and health is good but I can't DELETE or > CLOSE index as it is unresponsive. > > mlockall is set to true > > iostat: > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.00 0.05 0.30 0.08 0.00 97.57 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 7.40 0.00 939.20 0 4696 > sda 0.40 0.00 4.80 0 24 > dm-0 0.60 0.00 4.80 0 24 > dm-1 0.00 0.00 0.00 0 0 > dm-2 117.40 0.00 939.20 0 4696 > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.93 0.03 0.23 0.08 0.00 96.74 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 6.80 0.00 776.00 0 3880 > sda 0.80 0.00 20.80 0 104 > dm-0 2.60 0.00 20.80 0 104 > dm-1 0.00 0.00 0.00 0 0 > dm-2 97.00 0.00 776.00 0 3880 > > avg-cpu: %user %nice %system %iowait %steal %idle > 1.20 0.03 0.25 0.10 0.00 98.42 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 11.40 0.00 1312.00 0 6560 > sda 0.80 0.00 22.40 0 112 > dm-0 2.80 0.00 22.40 0 112 > dm-1 0.00 0.00 0.00 0 0 > dm-2 164.00 0.00 1312.00 0 6560 > > avg-cpu: %user %nice %system %iowait %steal %idle > 7.07 0.03 0.50 0.08 0.00 92.33 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 20.40 0.00 5064.00 0 25320 > sda 1.00 0.00 25.60 0 128 > dm-0 3.20 0.00 25.60 0 128 > dm-1 0.00 0.00 0.00 0 0 > dm-2 633.00 0.00 5064.00 0 25320 > > avg-cpu: %user %nice %system %iowait %steal %idle > 1.23 0.05 0.33 0.10 0.00 98.30 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 15.20 0.00 2604.80 0 13024 > sda 2.40 0.00 38.40 0 192 > dm-0 4.80 0.00 38.40 0 192 > dm-1 0.00 0.00 0.00 0 0 > dm-2 325.60 0.00 2604.80 0 13024 > > vmstat: > > -bash-4.1$ vmstat 5 > procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu----- > r b swpd free buff cache si so bi bo in cs us sy id > wa st > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > 96 2 0 > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > 97 0 0 > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > 98 0 0 > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > 95 0 0 > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > 95 0 0 > > -bash-4.1$ lsof -u elasticsearch | wc -l > 3004 > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > /etc/security/limits.conf:elasticsearch soft nofile 65536 > /etc/security/limits.conf:elasticsearch - memlock unlimited > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > Swap: 0k total, 0k used, 0k free, 1963024k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bXT8YJq%2B-qXgrPEo2NHJrjOzHtHw%2BTje-a08R6\_XCE6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bXT8YJq%2B-qXgrPEo2NHJrjOzHtHw%2BTje-a08R6_XCE6w%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 24, 2014, 10:40pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/3 "2014-09-24T22:40:21Z")
Green On Wed, Sep 24, 2014 at 6:20 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote: > What state is your cluster in? Can you get \_cat/health to return? > Regards, > Mark Walkom > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > On 25 September 2014 08:16, Chris Denneen [cdenneen@gmail.com](mailto:cdenneen@gmail.com) wrote: > > > If anyone can help me understand why my cluster is hung I would appreciate > > it. > > > > jstack output: > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > I am able to query the cluster and health is good but I can't DELETE or > > CLOSE index as it is unresponsive. > > > > mlockall is set to true > > > > iostat: > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 7.40 0.00 939.20 0 4696 > > sda 0.40 0.00 4.80 0 24 > > dm-0 0.60 0.00 4.80 0 24 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 117.40 0.00 939.20 0 4696 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 6.80 0.00 776.00 0 3880 > > sda 0.80 0.00 20.80 0 104 > > dm-0 2.60 0.00 20.80 0 104 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 97.00 0.00 776.00 0 3880 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 11.40 0.00 1312.00 0 6560 > > sda 0.80 0.00 22.40 0 112 > > dm-0 2.80 0.00 22.40 0 112 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 164.00 0.00 1312.00 0 6560 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 20.40 0.00 5064.00 0 25320 > > sda 1.00 0.00 25.60 0 128 > > dm-0 3.20 0.00 25.60 0 128 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 633.00 0.00 5064.00 0 25320 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 15.20 0.00 2604.80 0 13024 > > sda 2.40 0.00 38.40 0 192 > > dm-0 4.80 0.00 38.40 0 192 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 325.60 0.00 2604.80 0 13024 > > > > vmstat: > > > > -bash-4.1$ vmstat 5 > > procs -----------memory---------- ---swap-- -----io---- --system-- > > -----cpu----- > > r b swpd free buff cache si so bi bo in cs us sy id > > wa st > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > > 96 2 0 > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > > 97 0 0 > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > > 98 0 0 > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > > 95 0 0 > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > > 95 0 0 > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > 3004 > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/VUpE88PNFvI/unsubscribe](https://groups.google.com/d/topic/elasticsearch/VUpE88PNFvI/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bXT8YJq%2B-qXgrPEo2NHJrjOzHtHw%2BTje-a08R6\_XCE6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bXT8YJq%2B-qXgrPEo2NHJrjOzHtHw%2BTje-a08R6_XCE6w%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1411598421315.815642f3%40Nodemailer](https://groups.google.com/d/msgid/elasticsearch/1411598421315.815642f3%40Nodemailer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 24, 2014, 10:43pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/4 "2014-09-24T22:43:22Z")
I can even use HQ, paramedic and head plugins. On Wed, Sep 24, 2014 at 6:20 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote: > What state is your cluster in? Can you get \_cat/health to return? > Regards, > Mark Walkom > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > On 25 September 2014 08:16, Chris Denneen [cdenneen@gmail.com](mailto:cdenneen@gmail.com) wrote: > > > If anyone can help me understand why my cluster is hung I would appreciate > > it. > > > > jstack output: > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > I am able to query the cluster and health is good but I can't DELETE or > > CLOSE index as it is unresponsive. > > > > mlockall is set to true > > > > iostat: > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 7.40 0.00 939.20 0 4696 > > sda 0.40 0.00 4.80 0 24 > > dm-0 0.60 0.00 4.80 0 24 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 117.40 0.00 939.20 0 4696 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 6.80 0.00 776.00 0 3880 > > sda 0.80 0.00 20.80 0 104 > > dm-0 2.60 0.00 20.80 0 104 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 97.00 0.00 776.00 0 3880 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 11.40 0.00 1312.00 0 6560 > > sda 0.80 0.00 22.40 0 112 > > dm-0 2.80 0.00 22.40 0 112 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 164.00 0.00 1312.00 0 6560 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 20.40 0.00 5064.00 0 25320 > > sda 1.00 0.00 25.60 0 128 > > dm-0 3.20 0.00 25.60 0 128 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 633.00 0.00 5064.00 0 25320 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 15.20 0.00 2604.80 0 13024 > > sda 2.40 0.00 38.40 0 192 > > dm-0 4.80 0.00 38.40 0 192 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 325.60 0.00 2604.80 0 13024 > > > > vmstat: > > > > -bash-4.1$ vmstat 5 > > procs -----------memory---------- ---swap-- -----io---- --system-- > > -----cpu----- > > r b swpd free buff cache si so bi bo in cs us sy id > > wa st > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > > 96 2 0 > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > > 97 0 0 > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > > 98 0 0 > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > > 95 0 0 > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > > 95 0 0 > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > 3004 > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/VUpE88PNFvI/unsubscribe](https://groups.google.com/d/topic/elasticsearch/VUpE88PNFvI/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bXT8YJq%2B-qXgrPEo2NHJrjOzHtHw%2BTje-a08R6\_XCE6w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bXT8YJq%2B-qXgrPEo2NHJrjOzHtHw%2BTje-a08R6_XCE6w%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1411598602648.8da91a11%40Nodemailer](https://groups.google.com/d/msgid/elasticsearch/1411598602648.8da91a11%40Nodemailer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 25, 2014, 3:05pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/5 "2014-09-25T15:05:49Z")
Is there anymore info I can provide for someone to help here, I'm not sure what to do other than restart ES but that isn't a good long term solution every day or so? [root@rndeslogs1 elasticsearch]# curl -q localhost:9200/\_cluster/health | python -mjson.tool % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 116 233 116 233 0 0 10457 0 --:--:-- --:--:-- --:--:-- 13705 { "active\_primary\_shards": 136, "active\_shards": 136, "cluster\_name": "logstash-cluster", "initializing\_shards": 0, "number\_of\_data\_nodes": 1, "number\_of\_nodes": 2, "relocating\_shards": 0, "status": "yellow", _This is because I have marvel installed and only one data node but otherwise everything is green... when I DELETE .marvel_ indices cluster shows as "green" but because right now I can't DELETE, CLOSE, POST data to cluster it's showing as yellow\* "timed\_out": false, "unassigned\_shards": 12 } On Wednesday, September 24, 2014 6:16:51 PM UTC-4, Chris Denneen wrote: > If anyone can help me understand why my cluster is hung I would appreciate > it. > > jstack output: > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > I am able to query the cluster and health is good but I can't DELETE or > CLOSE index as it is unresponsive. > > mlockall is set to true > > iostat: > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.00 0.05 0.30 0.08 0.00 97.57 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 7.40 0.00 939.20 0 4696 > sda 0.40 0.00 4.80 0 24 > dm-0 0.60 0.00 4.80 0 24 > dm-1 0.00 0.00 0.00 0 0 > dm-2 117.40 0.00 939.20 0 4696 > > avg-cpu: %user %nice %system %iowait %steal %idle > 2.93 0.03 0.23 0.08 0.00 96.74 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 6.80 0.00 776.00 0 3880 > sda 0.80 0.00 20.80 0 104 > dm-0 2.60 0.00 20.80 0 104 > dm-1 0.00 0.00 0.00 0 0 > dm-2 97.00 0.00 776.00 0 3880 > > avg-cpu: %user %nice %system %iowait %steal %idle > 1.20 0.03 0.25 0.10 0.00 98.42 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 11.40 0.00 1312.00 0 6560 > sda 0.80 0.00 22.40 0 112 > dm-0 2.80 0.00 22.40 0 112 > dm-1 0.00 0.00 0.00 0 0 > dm-2 164.00 0.00 1312.00 0 6560 > > avg-cpu: %user %nice %system %iowait %steal %idle > 7.07 0.03 0.50 0.08 0.00 92.33 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 20.40 0.00 5064.00 0 25320 > sda 1.00 0.00 25.60 0 128 > dm-0 3.20 0.00 25.60 0 128 > dm-1 0.00 0.00 0.00 0 0 > dm-2 633.00 0.00 5064.00 0 25320 > > avg-cpu: %user %nice %system %iowait %steal %idle > 1.23 0.05 0.33 0.10 0.00 98.30 > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > sdb 15.20 0.00 2604.80 0 13024 > sda 2.40 0.00 38.40 0 192 > dm-0 4.80 0.00 38.40 0 192 > dm-1 0.00 0.00 0.00 0 0 > dm-2 325.60 0.00 2604.80 0 13024 > > vmstat: > > -bash-4.1$ vmstat 5 > procs -----------memory---------- ---swap-- -----io---- --system-- > -----cpu----- > r b swpd free buff cache si so bi bo in cs us sy id > wa st > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > 96 2 0 > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > 97 0 0 > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > 98 0 0 > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > 95 0 0 > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > 95 0 0 > > -bash-4.1$ lsof -u elasticsearch | wc -l > 3004 > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > /etc/security/limits.conf:elasticsearch soft nofile 65536 > /etc/security/limits.conf:elasticsearch - memlock unlimited > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > Swap: 0k total, 0k used, 0k free, 1963024k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 25, 2014, 3:21pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/6 "2014-09-25T15:21:40Z")
Mark, # curl -q localhost:9200/\_cat/health?pretty 1411658477 11:21:17 logstash-cluster yellow 2 1 136 136 0 0 12 On Wednesday, September 24, 2014 6:20:59 PM UTC-4, Mark Walkom wrote: > What state is your cluster in? Can you get \_cat/health to return? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 25 September 2014 08:16, Chris Denneen \<[cden...@gmail.com](mailto:cden...@gmail.com) \ > > > wrote: > > > If anyone can help me understand why my cluster is hung I would > > appreciate it. > > > > jstack output: > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > I am able to query the cluster and health is good but I can't DELETE or > > CLOSE index as it is unresponsive. > > > > mlockall is set to true > > > > iostat: > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 7.40 0.00 939.20 0 4696 > > sda 0.40 0.00 4.80 0 24 > > dm-0 0.60 0.00 4.80 0 24 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 117.40 0.00 939.20 0 4696 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 6.80 0.00 776.00 0 3880 > > sda 0.80 0.00 20.80 0 104 > > dm-0 2.60 0.00 20.80 0 104 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 97.00 0.00 776.00 0 3880 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 11.40 0.00 1312.00 0 6560 > > sda 0.80 0.00 22.40 0 112 > > dm-0 2.80 0.00 22.40 0 112 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 164.00 0.00 1312.00 0 6560 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 20.40 0.00 5064.00 0 25320 > > sda 1.00 0.00 25.60 0 128 > > dm-0 3.20 0.00 25.60 0 128 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 633.00 0.00 5064.00 0 25320 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 15.20 0.00 2604.80 0 13024 > > sda 2.40 0.00 38.40 0 192 > > dm-0 4.80 0.00 38.40 0 192 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 325.60 0.00 2604.80 0 13024 > > > > vmstat: > > > > -bash-4.1$ vmstat 5 > > procs -----------memory---------- ---swap-- -----io---- --system-- > > -----cpu----- > > r b swpd free buff cache si so bi bo in cs us sy > > id wa st > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > > 96 2 0 > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > > 97 0 0 > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > > 98 0 0 > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > > 95 0 0 > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > > 95 0 0 > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > 3004 > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/030f2841-a538-4276-8b2e-859230050bad%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/030f2841-a538-4276-8b2e-859230050bad%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 25, 2014, 3:54pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/7 "2014-09-25T15:54:43Z")
Check your log4j appenders. They block and ES can't continue. Jörg On Thu, Sep 25, 2014 at 5:05 PM, Chris Denneen [cdenneen@gmail.com](mailto:cdenneen@gmail.com) wrote: > Is there anymore info I can provide for someone to help here, I'm not sure > what to do other than restart ES but that isn't a good long term solution > every day or so? > > [root@rndeslogs1 elasticsearch]# curl -q localhost:9200/\_cluster/health | > python -mjson.tool > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 116 233 116 233 0 0 10457 0 --:--:-- --:--:-- --:--:-- > 13705 > { > "active\_primary\_shards": 136, > "active\_shards": 136, > "cluster\_name": "logstash-cluster", > "initializing\_shards": 0, > "number\_of\_data\_nodes": 1, > "number\_of\_nodes": 2, > "relocating\_shards": 0, > "status": "yellow", _This is because I have marvel installed and only > one data node but otherwise everything is green... when I DELETE .marvel_ > indices cluster shows as "green" but because right now I can't DELETE, > CLOSE, POST data to cluster it's showing as yellow\* > "timed\_out": false, > "unassigned\_shards": 12 > } > > On Wednesday, September 24, 2014 6:16:51 PM UTC-4, Chris Denneen wrote: > > > If anyone can help me understand why my cluster is hung I would > > appreciate it. > > > > jstack output: > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > I am able to query the cluster and health is good but I can't DELETE or > > CLOSE index as it is unresponsive. > > > > mlockall is set to true > > > > iostat: > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 7.40 0.00 939.20 0 4696 > > sda 0.40 0.00 4.80 0 24 > > dm-0 0.60 0.00 4.80 0 24 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 117.40 0.00 939.20 0 4696 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 6.80 0.00 776.00 0 3880 > > sda 0.80 0.00 20.80 0 104 > > dm-0 2.60 0.00 20.80 0 104 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 97.00 0.00 776.00 0 3880 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 11.40 0.00 1312.00 0 6560 > > sda 0.80 0.00 22.40 0 112 > > dm-0 2.80 0.00 22.40 0 112 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 164.00 0.00 1312.00 0 6560 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 20.40 0.00 5064.00 0 25320 > > sda 1.00 0.00 25.60 0 128 > > dm-0 3.20 0.00 25.60 0 128 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 633.00 0.00 5064.00 0 25320 > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > sdb 15.20 0.00 2604.80 0 13024 > > sda 2.40 0.00 38.40 0 192 > > dm-0 4.80 0.00 38.40 0 192 > > dm-1 0.00 0.00 0.00 0 0 > > dm-2 325.60 0.00 2604.80 0 13024 > > > > vmstat: > > > > -bash-4.1$ vmstat 5 > > procs -----------memory---------- ---swap-- -----io---- --system-- > > -----cpu----- > > r b swpd free buff cache si so bi bo in cs us sy > > id wa st > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > > 96 2 0 > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > > 97 0 0 > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > > 98 0 0 > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > > 95 0 0 > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > > 95 0 0 > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > 3004 > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, 0.32 > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > 0.0%st > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGJPK3xff7YiVjXVHcodGhh8wZfqhNqdVwReEAEDa%2BjQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHGJPK3xff7YiVjXVHcodGhh8wZfqhNqdVwReEAEDa%2BjQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 25, 2014, 4:02pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/8 "2014-09-25T16:02:32Z")
> > > There are more than three files. show original On Thursday, September 25, 2014 11:21:40 AM UTC-4, Chris Denneen wrote: > Mark, > > # curl -q localhost:9200/\_cat/health?pretty > > 1411658477 11:21:17 logstash-cluster yellow 2 1 136 136 0 0 12 > > On Wednesday, September 24, 2014 6:20:59 PM UTC-4, Mark Walkom wrote: > > > What state is your cluster in? Can you get \_cat/health to return? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 25 September 2014 08:16, Chris Denneen [cden...@gmail.com](mailto:cden...@gmail.com) wrote: > > > > > If anyone can help me understand why my cluster is hung I would > > > appreciate it. > > > > > > jstack output: > > > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > > > I am able to query the cluster and health is good but I can't DELETE or > > > CLOSE index as it is unresponsive. > > > > > > mlockall is set to true > > > > > > iostat: > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 7.40 0.00 939.20 0 4696 > > > sda 0.40 0.00 4.80 0 24 > > > dm-0 0.60 0.00 4.80 0 24 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 117.40 0.00 939.20 0 4696 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 6.80 0.00 776.00 0 3880 > > > sda 0.80 0.00 20.80 0 104 > > > dm-0 2.60 0.00 20.80 0 104 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 97.00 0.00 776.00 0 3880 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 11.40 0.00 1312.00 0 6560 > > > sda 0.80 0.00 22.40 0 112 > > > dm-0 2.80 0.00 22.40 0 112 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 164.00 0.00 1312.00 0 6560 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 20.40 0.00 5064.00 0 25320 > > > sda 1.00 0.00 25.60 0 128 > > > dm-0 3.20 0.00 25.60 0 128 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 633.00 0.00 5064.00 0 25320 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 15.20 0.00 2604.80 0 13024 > > > sda 2.40 0.00 38.40 0 192 > > > dm-0 4.80 0.00 38.40 0 192 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 325.60 0.00 2604.80 0 13024 > > > > > > vmstat: > > > > > > -bash-4.1$ vmstat 5 > > > procs -----------memory---------- ---swap-- -----io---- --system-- > > > -----cpu----- > > > r b swpd free buff cache si so bi bo in cs us sy > > > id wa st > > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > > > 96 2 0 > > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > > > 97 0 0 > > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > > > 98 0 0 > > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > > > 95 0 0 > > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > > > 95 0 0 > > > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > > 3004 > > > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, > > > 0.32 > > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > > 0.0%st > > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aee7cbd8-da2d-47b5-bf82-22ef1f1805b0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4a4f2963-3983-40e2-a8f8-89b7cd677b11%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a4f2963-3983-40e2-a8f8-89b7cd677b11%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Chris\_Denneen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_denneen/32/1254_2.png) [@Chris\_Denneen](https://discuss.elastic.co/u/Chris_Denneen)\ **Post date:** [September 25, 2014, 4:05pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/9 "2014-09-25T16:05:54Z")
Jörg, I've updated gist ([elasticsearch.yml · GitHub](https://gist.github.com/cdenneen/70049c77fa5fc547428e)) with logging.yml And NC shows 9500 as open... rest are just local files: [root@rndeslogs1 elasticsearch]# nc -z 127.0.0.1 9500 Connection to 127.0.0.1 9500 port [tcp/ismserver] succeeded! [root@rndeslogs1 elasticsearch]# nc -z localhost 9500 Connection to localhost 9500 port [tcp/ismserver] succeeded! -Chris On Thursday, September 25, 2014 11:54:56 AM UTC-4, Jörg Prante wrote: > Check your log4j appenders. They block and ES can't continue. > > Jörg > > On Thu, Sep 25, 2014 at 5:05 PM, Chris Denneen \<[cden...@gmail.com](mailto:cden...@gmail.com) > \\> wrote: > > > Is there anymore info I can provide for someone to help here, I'm not > > sure what to do other than restart ES but that isn't a good long term > > solution every day or so? > > > > [root@rndeslogs1 elasticsearch]# curl -q localhost:9200/\_cluster/health | > > python -mjson.tool > > % Total % Received % Xferd Average Speed Time Time Time > > Current > > Dload Upload Total Spent Left > > Speed > > 116 233 116 233 0 0 10457 0 --:--:-- --:--:-- --:--:-- > > 13705 > > { > > "active\_primary\_shards": 136, > > "active\_shards": 136, > > "cluster\_name": "logstash-cluster", > > "initializing\_shards": 0, > > "number\_of\_data\_nodes": 1, > > "number\_of\_nodes": 2, > > "relocating\_shards": 0, > > "status": "yellow", _This is because I have marvel installed and > > only one data node but otherwise everything is green... when I DELETE > > .marvel_ indices cluster shows as "green" but because right now I can't > > DELETE, CLOSE, POST data to cluster it's showing as yellow\* > > "timed\_out": false, > > "unassigned\_shards": 12 > > } > > > > On Wednesday, September 24, 2014 6:16:51 PM UTC-4, Chris Denneen wrote: > > > > > If anyone can help me understand why my cluster is hung I would > > > appreciate it. > > > > > > jstack output: > > > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > > > I am able to query the cluster and health is good but I can't DELETE or > > > CLOSE index as it is unresponsive. > > > > > > mlockall is set to true > > > > > > iostat: > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 7.40 0.00 939.20 0 4696 > > > sda 0.40 0.00 4.80 0 24 > > > dm-0 0.60 0.00 4.80 0 24 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 117.40 0.00 939.20 0 4696 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 6.80 0.00 776.00 0 3880 > > > sda 0.80 0.00 20.80 0 104 > > > dm-0 2.60 0.00 20.80 0 104 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 97.00 0.00 776.00 0 3880 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 11.40 0.00 1312.00 0 6560 > > > sda 0.80 0.00 22.40 0 112 > > > dm-0 2.80 0.00 22.40 0 112 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 164.00 0.00 1312.00 0 6560 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 20.40 0.00 5064.00 0 25320 > > > sda 1.00 0.00 25.60 0 128 > > > dm-0 3.20 0.00 25.60 0 128 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 633.00 0.00 5064.00 0 25320 > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > sdb 15.20 0.00 2604.80 0 13024 > > > sda 2.40 0.00 38.40 0 192 > > > dm-0 4.80 0.00 38.40 0 192 > > > dm-1 0.00 0.00 0.00 0 0 > > > dm-2 325.60 0.00 2604.80 0 13024 > > > > > > vmstat: > > > > > > -bash-4.1$ vmstat 5 > > > procs -----------memory---------- ---swap-- -----io---- --system-- > > > -----cpu----- > > > r b swpd free buff cache si so bi bo in cs us sy > > > id wa st > > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 0 > > > 96 2 0 > > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 0 > > > 97 0 0 > > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 0 > > > 98 0 0 > > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 0 > > > 95 0 0 > > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 0 > > > 95 0 0 > > > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > > 3004 > > > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, > > > 0.32 > > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > > 0.0%st > > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/50b8e6ef-8c32-4f66-919d-19bfd3cd4a43%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/50b8e6ef-8c32-4f66-919d-19bfd3cd4a43%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [September 30, 2014, 3:18pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/10 "2014-09-30T15:18:42Z")
Chris, This sounds very suspiciously like a problem we had. We set up an experimental local ELK server (one node in the cluster) and fed it with logstash. I was manually cleaning up older data using the Elasticsearch Head plug-in, but over one weekend the cluster got into a funky state. The curl API said it was Yellow, but ES Head showed Green, and queries were hanging. This was a VM that was dedicated to ES with 1TB disk space (only about 2% was ever used at any point in time), 4 CPUs, and 24GB RAM (though the Java JVM was not tuned to take advantage of all of this memory). Kibana was hosted as a site plug-in, but its usage was very light. Though I had been playing around with increasing the size limit of responses way past the default of 500, and I'm sure the ES server bore the brunt of that. I stopped and restarted ES and everything went back to normal. I installed Curator to clean up older indices automatically, and the problem has never returned. (I have also stopped telling Kibana to ask for up to 50000 response documents on a query!) I suspect you're getting some sort of OOM condition and that's when things start looking odd. Anyway, OOM is just a wild guess. I wouldn't have mentioned something so nebulous, but the symptoms you have are strikingly close to the ones we saw. Brian -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ccf8ab3d-c89c-42da-95ba-1b25198fc445%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ccf8ab3d-c89c-42da-95ba-1b25198fc445%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 30, 2014, 7:05pm UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/11 "2014-09-30T19:05:11Z")
You have a socket appender which blocks, and this stalls ES. Maybe you use TCP and not UDP. UDB can not block. This has been improved in log4j2 where socketappender can be configured as an async appender which never blocks, even with TCP. Check if you can switch to log4j2: [http://logging.apache.org/log4j/2.x/manual/appenders.html](http://logging.apache.org/log4j/2.x/manual/appenders.html) Jörg socketappender: type: org.apache.log4j.net.SocketAppender port: 9500 remoteHost: localhost layout: type: pattern conversionPattern: "[%d{ISO8601}][%-5p][%-25c] %m%n" On Thu, Sep 25, 2014 at 6:05 PM, Chris Denneen [cdenneen@gmail.com](mailto:cdenneen@gmail.com) wrote: > Jörg, > > I've updated gist ([elasticsearch.yml · GitHub](https://gist.github.com/cdenneen/70049c77fa5fc547428e)) > with logging.yml > > And NC shows 9500 as open... rest are just local files: > > [root@rndeslogs1 elasticsearch]# nc -z 127.0.0.1 9500 > Connection to 127.0.0.1 9500 port [tcp/ismserver] succeeded! > [root@rndeslogs1 elasticsearch]# nc -z localhost 9500 > Connection to localhost 9500 port [tcp/ismserver] succeeded! > > -Chris > > On Thursday, September 25, 2014 11:54:56 AM UTC-4, Jörg Prante wrote: > > > Check your log4j appenders. They block and ES can't continue. > > > > Jörg > > > > On Thu, Sep 25, 2014 at 5:05 PM, Chris Denneen [cden...@gmail.com](mailto:cden...@gmail.com) wrote: > > > > > Is there anymore info I can provide for someone to help here, I'm not > > > sure what to do other than restart ES but that isn't a good long term > > > solution every day or so? > > > > > > [root@rndeslogs1 elasticsearch]# curl -q localhost:9200/\_cluster/health > > > | python -mjson.tool > > > % Total % Received % Xferd Average Speed Time Time Time > > > Current > > > Dload Upload Total Spent Left > > > Speed > > > 116 233 116 233 0 0 10457 0 --:--:-- --:--:-- --:--:-- > > > 13705 > > > { > > > "active\_primary\_shards": 136, > > > "active\_shards": 136, > > > "cluster\_name": "logstash-cluster", > > > "initializing\_shards": 0, > > > "number\_of\_data\_nodes": 1, > > > "number\_of\_nodes": 2, > > > "relocating\_shards": 0, > > > "status": "yellow", _This is because I have marvel installed and > > > only one data node but otherwise everything is green... when I DELETE > > > .marvel_ indices cluster shows as "green" but because right now I can't > > > DELETE, CLOSE, POST data to cluster it's showing as yellow\* > > > "timed\_out": false, > > > "unassigned\_shards": 12 > > > } > > > > > > On Wednesday, September 24, 2014 6:16:51 PM UTC-4, Chris Denneen wrote: > > > > > > > If anyone can help me understand why my cluster is hung I would > > > > appreciate it. > > > > > > > > jstack output: > > > > > > > > [Cist created gist · GitHub](https://gist.github.com/anonymous/075c862cb211ae249707) > > > > > > > > I am able to query the cluster and health is good but I can't DELETE or > > > > CLOSE index as it is unresponsive. > > > > > > > > mlockall is set to true > > > > > > > > iostat: > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > 2.00 0.05 0.30 0.08 0.00 97.57 > > > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > > sdb 7.40 0.00 939.20 0 4696 > > > > sda 0.40 0.00 4.80 0 24 > > > > dm-0 0.60 0.00 4.80 0 24 > > > > dm-1 0.00 0.00 0.00 0 0 > > > > dm-2 117.40 0.00 939.20 0 4696 > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > 2.93 0.03 0.23 0.08 0.00 96.74 > > > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > > sdb 6.80 0.00 776.00 0 3880 > > > > sda 0.80 0.00 20.80 0 104 > > > > dm-0 2.60 0.00 20.80 0 104 > > > > dm-1 0.00 0.00 0.00 0 0 > > > > dm-2 97.00 0.00 776.00 0 3880 > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > 1.20 0.03 0.25 0.10 0.00 98.42 > > > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > > sdb 11.40 0.00 1312.00 0 6560 > > > > sda 0.80 0.00 22.40 0 112 > > > > dm-0 2.80 0.00 22.40 0 112 > > > > dm-1 0.00 0.00 0.00 0 0 > > > > dm-2 164.00 0.00 1312.00 0 6560 > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > 7.07 0.03 0.50 0.08 0.00 92.33 > > > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > > sdb 20.40 0.00 5064.00 0 25320 > > > > sda 1.00 0.00 25.60 0 128 > > > > dm-0 3.20 0.00 25.60 0 128 > > > > dm-1 0.00 0.00 0.00 0 0 > > > > dm-2 633.00 0.00 5064.00 0 25320 > > > > > > > > avg-cpu: %user %nice %system %iowait %steal %idle > > > > 1.23 0.05 0.33 0.10 0.00 98.30 > > > > > > > > Device: tps Blk\_read/s Blk\_wrtn/s Blk\_read Blk\_wrtn > > > > sdb 15.20 0.00 2604.80 0 13024 > > > > sda 2.40 0.00 38.40 0 192 > > > > dm-0 4.80 0.00 38.40 0 192 > > > > dm-1 0.00 0.00 0.00 0 0 > > > > dm-2 325.60 0.00 2604.80 0 13024 > > > > > > > > vmstat: > > > > > > > > -bash-4.1$ vmstat 5 > > > > procs -----------memory---------- ---swap-- -----io---- --system-- > > > > -----cpu----- > > > > r b swpd free buff cache si so bi bo in cs us sy > > > > id wa st > > > > 0 0 0 141532 163140 1955776 0 0 19 80 2 0 2 > > > > 0 96 2 0 > > > > 0 0 0 140664 163156 1956428 0 0 0 801 776 719 3 > > > > 0 97 0 0 > > > > 0 0 0 138880 163164 1958264 0 0 0 776 770 765 2 > > > > 0 98 0 0 > > > > 0 0 0 133820 163192 1963364 0 0 0 1570 1174 825 4 > > > > 0 95 0 0 > > > > 1 0 0 129984 163200 1967036 0 0 0 1422 1026 836 4 > > > > 0 95 0 0 > > > > > > > > -bash-4.1$ lsof -u elasticsearch | wc -l > > > > 3004 > > > > > > > > /etc/security/limits.conf:elasticsearch hard nofile 65536 > > > > /etc/security/limits.conf:elasticsearch soft nofile 65536 > > > > /etc/security/limits.conf:elasticsearch - memlock unlimited > > > > > > > > top - 18:15:25 up 18 days, 14:36, 1 user, load average: 0.23, 0.32, > > > > 0.32 > > > > Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie > > > > Cpu(s): 0.5%us, 0.2%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, > > > > 0.0%st > > > > Mem: 8060812k total, 7928472k used, 132340k free, 164384k buffers > > > > Swap: 0k total, 0k used, 0k free, 1963024k cached > > > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > > > > 26117 elastics 20 0 55.0g 5.2g 327m S 4.3 68.1 1836:21 java > > > > 1358 logstash 39 19 5078m 257m 11m S 0.7 3.3 183:28.43 java > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/77017349-b637-450f-8923-7e27c8bfa8d0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/50b8e6ef-8c32-4f66-919d-19bfd3cd4a43%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/50b8e6ef-8c32-4f66-919d-19bfd3cd4a43%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/50b8e6ef-8c32-4f66-919d-19bfd3cd4a43%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/50b8e6ef-8c32-4f66-919d-19bfd3cd4a43%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEjEGWxD2hYRcHuaF0zCfXNC-0wAGpNGDG43\_OS5mUYgg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEjEGWxD2hYRcHuaF0zCfXNC-0wAGpNGDG43_OS5mUYgg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:59am UTC](https://discuss.elastic.co/t/elasticsearch-blocked-futex/19953/12 "2017-07-06T00:59:04Z")