# Fmutex hanging issue with elasticsearch + rabbitmq river + large dataset (\>100M records) **URL:** \ **Category:** Elasticsearch\ **Created:** [April 16, 2012, 11:40pm UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361 "2012-04-16T23:40:46Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 16, 2012, 11:40pm UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/1 "2012-04-16T23:40:46Z")
Everything was chugging along, processing about 2k messages/sec and then just hanged. Running strace on the process shows the trace below, which persists even through restarts of elasticsearch. It seems that the issue occurs right after a clone system call. Ideas? * * * open("/usr/share/java/elasticsearch/lib/sigar/", O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 getdents(3, /\* 4 entries _/, 32768) = 136 getdents(3, /_ 0 entries \*/, 32768) = 0 close(3) = 0 mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 clone(child\_stack=0x7f4c05f98ff0, flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, child\_tidptr=0x7f4c05f999d0) = 32755 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be restarted) --- SIGINT (Interrupt) @ 0 (0) --- futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 rt\_sigreturn(0x7f4c06c23440) = 202 futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 17, 2012, 1:20pm UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/2 "2012-04-17T13:20:10Z")
Can you issue a jstack and gist the stack trace you get? On Tue, Apr 17, 2012 at 2:40 AM, arimus [arimus@gmail.com](mailto:arimus@gmail.com) wrote: > Everything was chugging along, processing about 2k messages/sec and then > just hanged. Running strace on the process shows the trace below, which > persists even through restarts of elasticsearch. It seems that the issue > occurs right after a clone system call. Ideas? > > * * * > > open("/usr/share/java/elasticsearch/lib/sigar/", > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > getdents(3, /\* 4 entries _/, 32768) = 136 > getdents(3, /_ 0 entries \*/, 32768) = 0 > close(3) = 0 > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > clone(child\_stack=0x7f4c05f98ff0, > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > child\_tidptr=0x7f4c05f999d0) = 32755 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > restarted) > --- SIGINT (Interrupt) @ 0 (0) --- > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > rt\_sigreturn(0x7f4c06c23440) = 202 > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 19, 2012, 12:43am UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/3 "2012-04-19T00:43:23Z")
I ended up doing the following: - upgrading from rabbitmq-server-2.6.1-1.el6 to rabbitmq-server-2.8.1-1 - disabling a couple other elasticsearch instances that were trying to multicast discover on the same subnet - restarting everything So far, another 100M+ messages processed without issues, but I will do the jstack if I see the issue again. Looks like there was a post ([Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/A6Zlgzwaui4)) yesterday that was also futex-related, on debian and without the river though. Perhaps I just haven't run into it again yet. Thanks, will keep you posted. On Tuesday, April 17, 2012 6:20:10 AM UTC-7, kimchy wrote: > Can you issue a jstack and gist the stack trace you get? > > On Tue, Apr 17, 2012 at 2:40 AM, arimus wrote: > > > Everything was chugging along, processing about 2k messages/sec and then > > just hanged. Running strace on the process shows the trace below, which > > persists even through restarts of elasticsearch. It seems that the issue > > occurs right after a clone system call. Ideas? > > > > * * * > > > > open("/usr/share/java/elasticsearch/lib/sigar/", > > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > > getdents(3, /\* 4 entries _/, 32768) = 136 > > getdents(3, /_ 0 entries \*/, 32768) = 0 > > close(3) = 0 > > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > > clone(child\_stack=0x7f4c05f98ff0, > > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > > child\_tidptr=0x7f4c05f999d0) = 32755 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > restarted) > > --- SIGINT (Interrupt) @ 0 (0) --- > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > rt\_sigreturn(0x7f4c06c23440) = 202 > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 19, 2012, 7:27pm UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/4 "2012-04-19T19:27:18Z")
Alright, it finally died again and here is the jstack: [es.jstack · GitHub](https://gist.github.com/2423439) I'm keeping it hanged until we get to the bottom of this, so I'm willing to jump in IRC and pow-wow on finding a solution to get things running again. Just let me know. On Wednesday, April 18, 2012 5:43:23 PM UTC-7, arimus wrote: > I ended up doing the following: > > - upgrading from rabbitmq-server-2.6.1-1.el6 to rabbitmq-server-2.8.1-1 > - disabling a couple other elasticsearch instances that were trying to > multicast discover on the same subnet > - restarting everything > > So far, another 100M+ messages processed without issues, but I will do the > jstack if I see the issue again. Looks like there was a post ( > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/A6Zlgzwaui4)) yesterday > that was also futex-related, on debian and without the river though. > Perhaps I just haven't run into it again yet. Thanks, will keep you > posted. > > On Tuesday, April 17, 2012 6:20:10 AM UTC-7, kimchy wrote: > > > Can you issue a jstack and gist the stack trace you get? > > > > On Tue, Apr 17, 2012 at 2:40 AM, arimus wrote: > > > > > Everything was chugging along, processing about 2k messages/sec and then > > > just hanged. Running strace on the process shows the trace below, which > > > persists even through restarts of elasticsearch. It seems that the issue > > > occurs right after a clone system call. Ideas? > > > > > > * * * > > > > > > open("/usr/share/java/elasticsearch/lib/sigar/", > > > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > > > getdents(3, /\* 4 entries _/, 32768) = 136 > > > getdents(3, /_ 0 entries \*/, 32768) = 0 > > > close(3) = 0 > > > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > > > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > > > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > > > clone(child\_stack=0x7f4c05f98ff0, > > > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > > > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > > > child\_tidptr=0x7f4c05f999d0) = 32755 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > restarted) > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 19, 2012, 11:38pm UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/5 "2012-04-19T23:38:40Z")
Also, strace -f on the elasticsearch process shows lots of futex connection timeouts [https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt](https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt) On Thursday, April 19, 2012 12:27:18 PM UTC-7, arimus wrote: > Alright, it finally died again and here is the jstack: > > [es.jstack · GitHub](https://gist.github.com/2423439) > > I'm keeping it hanged until we get to the bottom of this, so I'm willing > to jump in IRC and pow-wow on finding a solution to get things running > again. Just let me know. > > On Wednesday, April 18, 2012 5:43:23 PM UTC-7, arimus wrote: > > > I ended up doing the following: > > > > - upgrading from rabbitmq-server-2.6.1-1.el6 > > to rabbitmq-server-2.8.1-1 > > - disabling a couple other elasticsearch instances that were trying > > to multicast discover on the same subnet > > - restarting everything > > > > So far, another 100M+ messages processed without issues, but I will do > > the jstack if I see the issue again. Looks like there was a post ( > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/A6Zlgzwaui4)) yesterday > > that was also futex-related, on debian and without the river though. > > Perhaps I just haven't run into it again yet. Thanks, will keep you > > posted. > > > > On Tuesday, April 17, 2012 6:20:10 AM UTC-7, kimchy wrote: > > > > > Can you issue a jstack and gist the stack trace you get? > > > > > > On Tue, Apr 17, 2012 at 2:40 AM, arimus wrote: > > > > > > > Everything was chugging along, processing about 2k messages/sec and > > > > then just hanged. Running strace on the process shows the trace below, > > > > which persists even through restarts of elasticsearch. It seems that the > > > > issue occurs right after a clone system call. Ideas? > > > > > > > > * * * > > > > > > > > open("/usr/share/java/elasticsearch/lib/sigar/", > > > > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > > > > getdents(3, /\* 4 entries _/, 32768) = 136 > > > > getdents(3, /_ 0 entries \*/, 32768) = 0 > > > > close(3) = 0 > > > > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > > > > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > > > > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > > > > clone(child\_stack=0x7f4c05f98ff0, > > > > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > > > > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > > > > child\_tidptr=0x7f4c05f999d0) = 32755 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > restarted) > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 20, 2012, 3:31am UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/6 "2012-04-20T03:31:23Z")
Well, decided to profile elasticsearch with YourKit to see what I could see. Looking at threads that appear blocked... Lots of org.apache.log4j.spi.RootLogger and org.elasticsearch.common.jackson.sym.BytesToNameCanonicalizer blocking on the elasticsearch[bulk] thread. So is it blocking on the elasticsearch river plugin then, or does it look like this just because that's 99% of the work that my instance is doing...and there is an underlying issue blocking things in the middle of regular processing? Seems likely that it's the later. > **[Skitch | Evernote](https://evernote.com/products/skitch)** > > Get your point across with fewer words using annotation, shapes and sketches, so that your ideas become reality faster. Interestingly, there are also a bunch of threads which appear blocked at java.security.SecureRandom. I have seen reference to random device entropy being a root cause for some java futex wait hang-ups ([http://www.springone2gx.com/blog/pratik\_patel/2010/01/solution\_futex\_wait\_hangs\_java\_on\_linux\_\_ubuntu\_in\_vmware\_or\_virtual\_box](http://www.springone2gx.com/blog/pratik_patel/2010/01/solution_futex_wait_hangs_java_on_linux__ubuntu_in_vmware_or_virtual_box)). Could be another red herring, but figured I would mention it. Perhaps the UUID generation for elasticsearch is depleting the SecureRandom entropy. > **[Skitch | Evernote](https://evernote.com/products/skitch)** > > Get your point across with fewer words using annotation, shapes and sketches, so that your ideas become reality faster. I'm going to give /dev/urandom a go for now and see if the issue goes away (e.g. -Djava.security.egd=file:/dev/urandom). On Thursday, April 19, 2012 4:38:40 PM UTC-7, arimus wrote: > Also, strace -f on the elasticsearch process shows lots of futex > connection timeouts > > [https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt](https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt) > > On Thursday, April 19, 2012 12:27:18 PM UTC-7, arimus wrote: > > > Alright, it finally died again and here is the jstack: > > > > [es.jstack · GitHub](https://gist.github.com/2423439) > > > > I'm keeping it hanged until we get to the bottom of this, so I'm willing > > to jump in IRC and pow-wow on finding a solution to get things running > > again. Just let me know. > > > > On Wednesday, April 18, 2012 5:43:23 PM UTC-7, arimus wrote: > > > > > I ended up doing the following: > > > > > > - upgrading from rabbitmq-server-2.6.1-1.el6 > > > to rabbitmq-server-2.8.1-1 > > > - disabling a couple other elasticsearch instances that were trying > > > to multicast discover on the same subnet > > > - restarting everything > > > > > > So far, another 100M+ messages processed without issues, but I will do > > > the jstack if I see the issue again. Looks like there was a post ( > > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/A6Zlgzwaui4)) yesterday > > > that was also futex-related, on debian and without the river though. > > > Perhaps I just haven't run into it again yet. Thanks, will keep you > > > posted. > > > > > > On Tuesday, April 17, 2012 6:20:10 AM UTC-7, kimchy wrote: > > > > > > > Can you issue a jstack and gist the stack trace you get? > > > > > > > > On Tue, Apr 17, 2012 at 2:40 AM, arimus wrote: > > > > > > > > > Everything was chugging along, processing about 2k messages/sec and > > > > > then just hanged. Running strace on the process shows the trace below, > > > > > which persists even through restarts of elasticsearch. It seems that the > > > > > issue occurs right after a clone system call. Ideas? > > > > > > > > > > * * * > > > > > > > > > > open("/usr/share/java/elasticsearch/lib/sigar/", > > > > > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > > > > > getdents(3, /\* 4 entries _/, 32768) = 136 > > > > > getdents(3, /_ 0 entries \*/, 32768) = 0 > > > > > close(3) = 0 > > > > > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > > > > > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > > > > > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > > > > > clone(child\_stack=0x7f4c05f98ff0, > > > > > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > > > > > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > > > > > child\_tidptr=0x7f4c05f999d0) = 32755 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > restarted) > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 20, 2012, 11:15pm UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/7 "2012-04-20T23:15:16Z")
So far, making this change and upping the elasticsearch min/max heap size seems to be helping. Going to reset everything and let it soak over the weekend to see if I can get it to run solid for a few days without any hiccups. Still seeing some index related issues on restart though. I've logged that issue as a bug ([Index issues on restarts with defaults · Issue #1871 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1871)) and will see if it crops up again after resetting everything. On Thursday, April 19, 2012 8:31:23 PM UTC-7, arimus wrote: > Well, decided to profile elasticsearch with YourKit to see what I could > see. Looking at threads that appear blocked... > > Lots of org.apache.log4j.spi.RootLogger and > org.elasticsearch.common.jackson.sym.BytesToNameCanonicalizer blocking on > the elasticsearch[bulk] thread. So is it blocking on the elasticsearch > river plugin then, or does it look like this just because that's 99% of the > work that my instance is doing...and there is an underlying issue blocking > things in the middle of regular processing? Seems likely that it's the > later. > > [Skitch | Evernote](https://skitch.com/arimus/8wtcp/elasticsearch-2012-04-19.snapshot-users-arimus-snapshots-yourkit-java-profiler-11.0.1) > > Interestingly, there are also a bunch of threads which appear blocked > at java.security.SecureRandom. I have seen reference to random device > entropy being a root cause for some java futex wait hang-ups ( > [http://www.springone2gx.com/blog/pratik\_patel/2010/01/solution\_futex\_wait\_hangs\_java\_on\_linux\_\_ubuntu\_in\_vmware\_or\_virtual\_box](http://www.springone2gx.com/blog/pratik_patel/2010/01/solution_futex_wait_hangs_java_on_linux__ubuntu_in_vmware_or_virtual_box)). > Could be another red herring, but figured I would mention it. Perhaps the > UUID generation for elasticsearch is depleting the SecureRandom entropy. > > [Skitch | Evernote](https://skitch.com/arimus/8wtps/elasticsearch-2012-04-19.snapshot-users-arimus-snapshots-yourkit-java-profiler-11.0.1) > > I'm going to give /dev/urandom a go for now and see if the issue goes away > (e.g. -Djava.security.egd=file:/dev/urandom). > > On Thursday, April 19, 2012 4:38:40 PM UTC-7, arimus wrote: > > > Also, strace -f on the elasticsearch process shows lots of futex > > connection timeouts > > > > [https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt](https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt) > > > > On Thursday, April 19, 2012 12:27:18 PM UTC-7, arimus wrote: > > > > > Alright, it finally died again and here is the jstack: > > > > > > [es.jstack · GitHub](https://gist.github.com/2423439) > > > > > > I'm keeping it hanged until we get to the bottom of this, so I'm willing > > > to jump in IRC and pow-wow on finding a solution to get things running > > > again. Just let me know. > > > > > > On Wednesday, April 18, 2012 5:43:23 PM UTC-7, arimus wrote: > > > > > > > I ended up doing the following: > > > > > > > > - upgrading from rabbitmq-server-2.6.1-1.el6 > > > > to rabbitmq-server-2.8.1-1 > > > > - disabling a couple other elasticsearch instances that were trying > > > > to multicast discover on the same subnet > > > > - restarting everything > > > > > > > > So far, another 100M+ messages processed without issues, but I will do > > > > the jstack if I see the issue again. Looks like there was a post ( > > > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/A6Zlgzwaui4)) yesterday > > > > that was also futex-related, on debian and without the river though. > > > > Perhaps I just haven't run into it again yet. Thanks, will keep you > > > > posted. > > > > > > > > On Tuesday, April 17, 2012 6:20:10 AM UTC-7, kimchy wrote: > > > > > > > > > Can you issue a jstack and gist the stack trace you get? > > > > > > > > > > On Tue, Apr 17, 2012 at 2:40 AM, arimus wrote: > > > > > > > > > > > Everything was chugging along, processing about 2k messages/sec and > > > > > > then just hanged. Running strace on the process shows the trace below, > > > > > > which persists even through restarts of elasticsearch. It seems that the > > > > > > issue occurs right after a clone system call. Ideas? > > > > > > > > > > > > * * * > > > > > > > > > > > > open("/usr/share/java/elasticsearch/lib/sigar/", > > > > > > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > > > > > > getdents(3, /\* 4 entries _/, 32768) = 136 > > > > > > getdents(3, /_ 0 entries \*/, 32768) = 0 > > > > > > close(3) = 0 > > > > > > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > > > > > > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > > > > > > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > > > > > > clone(child\_stack=0x7f4c05f98ff0, > > > > > > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > > > > > > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > > > > > > child\_tidptr=0x7f4c05f999d0) = 32755 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![arimus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arimus/32/2853_2.png) [@arimus](https://discuss.elastic.co/u/arimus)\ **Post date:** [April 21, 2012, 3:55am UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/8 "2012-04-21T03:55:21Z")
Things did indeed seem happier with more memory and the /dev/urandom tweak from what I could tell. Deleted some old indexes and letting it continue to soak over the weekend. Will let folks know if I see the same issue again (I may just be delaying the problem) or if this resolves the issues more permanently. On Thursday, April 19, 2012 8:31:23 PM UTC-7, arimus wrote: > Well, decided to profile elasticsearch with YourKit to see what I could > see. Looking at threads that appear blocked... > > Lots of org.apache.log4j.spi.RootLogger and > org.elasticsearch.common.jackson.sym.BytesToNameCanonicalizer blocking on > the elasticsearch[bulk] thread. So is it blocking on the elasticsearch > river plugin then, or does it look like this just because that's 99% of the > work that my instance is doing...and there is an underlying issue blocking > things in the middle of regular processing? Seems likely that it's the > later. > > [Skitch | Evernote](https://skitch.com/arimus/8wtcp/elasticsearch-2012-04-19.snapshot-users-arimus-snapshots-yourkit-java-profiler-11.0.1) > > Interestingly, there are also a bunch of threads which appear blocked > at java.security.SecureRandom. I have seen reference to random device > entropy being a root cause for some java futex wait hang-ups ( > [http://www.springone2gx.com/blog/pratik\_patel/2010/01/solution\_futex\_wait\_hangs\_java\_on\_linux\_\_ubuntu\_in\_vmware\_or\_virtual\_box](http://www.springone2gx.com/blog/pratik_patel/2010/01/solution_futex_wait_hangs_java_on_linux__ubuntu_in_vmware_or_virtual_box)). > Could be another red herring, but figured I would mention it. Perhaps the > UUID generation for elasticsearch is depleting the SecureRandom entropy. > > [Skitch | Evernote](https://skitch.com/arimus/8wtps/elasticsearch-2012-04-19.snapshot-users-arimus-snapshots-yourkit-java-profiler-11.0.1) > > I'm going to give /dev/urandom a go for now and see if the issue goes away > (e.g. -Djava.security.egd=file:/dev/urandom). > > On Thursday, April 19, 2012 4:38:40 PM UTC-7, arimus wrote: > > > Also, strace -f on the elasticsearch process shows lots of futex > > connection timeouts > > > > [https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt](https://raw.github.com/gist/2424865/a31d666c2629963f2f274ecfdf4b7f5a3881fbe6/gistfile1.txt) > > > > On Thursday, April 19, 2012 12:27:18 PM UTC-7, arimus wrote: > > > > > Alright, it finally died again and here is the jstack: > > > > > > [es.jstack · GitHub](https://gist.github.com/2423439) > > > > > > I'm keeping it hanged until we get to the bottom of this, so I'm willing > > > to jump in IRC and pow-wow on finding a solution to get things running > > > again. Just let me know. > > > > > > On Wednesday, April 18, 2012 5:43:23 PM UTC-7, arimus wrote: > > > > > > > I ended up doing the following: > > > > > > > > - upgrading from rabbitmq-server-2.6.1-1.el6 > > > > to rabbitmq-server-2.8.1-1 > > > > - disabling a couple other elasticsearch instances that were trying > > > > to multicast discover on the same subnet > > > > - restarting everything > > > > > > > > So far, another 100M+ messages processed without issues, but I will do > > > > the jstack if I see the issue again. Looks like there was a post ( > > > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/A6Zlgzwaui4)) yesterday > > > > that was also futex-related, on debian and without the river though. > > > > Perhaps I just haven't run into it again yet. Thanks, will keep you > > > > posted. > > > > > > > > On Tuesday, April 17, 2012 6:20:10 AM UTC-7, kimchy wrote: > > > > > > > > > Can you issue a jstack and gist the stack trace you get? > > > > > > > > > > On Tue, Apr 17, 2012 at 2:40 AM, arimus wrote: > > > > > > > > > > > Everything was chugging along, processing about 2k messages/sec and > > > > > > then just hanged. Running strace on the process shows the trace below, > > > > > > which persists even through restarts of elasticsearch. It seems that the > > > > > > issue occurs right after a clone system call. Ideas? > > > > > > > > > > > > * * * > > > > > > > > > > > > open("/usr/share/java/elasticsearch/lib/sigar/", > > > > > > O\_RDONLY|O\_NONBLOCK|O\_DIRECTORY|O\_CLOEXEC) = 3 > > > > > > getdents(3, /\* 4 entries _/, 32768) = 136 > > > > > > getdents(3, /_ 0 entries \*/, 32768) = 0 > > > > > > close(3) = 0 > > > > > > mmap(NULL, 135168, PROT\_READ|PROT\_WRITE, > > > > > > MAP\_PRIVATE|MAP\_ANONYMOUS|MAP\_STACK, -1, 0) = 0x7f4c05f79000 > > > > > > mprotect(0x7f4c05f79000, 4096, PROT\_NONE) = 0 > > > > > > clone(child\_stack=0x7f4c05f98ff0, > > > > > > flags=CLONE\_VM|CLONE\_FS|CLONE\_FILES|CLONE\_SIGHAND|CLONE\_THREAD|CLONE\_SYSVSEM|CLONE\_SETTLS|CLONE\_PARENT\_SETTID|CLONE\_CHILD\_CLEARTID, > > > > > > parent\_tidptr=0x7f4c05f999d0, tls=0x7f4c05f99700, > > > > > > child\_tidptr=0x7f4c05f999d0) = 32755 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL) = ? ERESTARTSYS (To be > > > > > > restarted) > > > > > > --- SIGINT (Interrupt) @ 0 (0) --- > > > > > > futex(0x7f4c06c23440, FUTEX\_WAKE\_PRIVATE, 1) = 1 > > > > > > rt\_sigreturn(0x7f4c06c23440) = 202 > > > > > > futex(0x7f4c05f999d0, FUTEX\_WAIT, 32755, NULL ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:31am UTC](https://discuss.elastic.co/t/fmutex-hanging-issue-with-elasticsearch-rabbitmq-river-large-dataset-100m-records/7361/9 "2017-07-06T03:31:36Z")