# Elasticsearch run as a service dies on first request **URL:** \ **Category:** Elasticsearch\ **Created:** [November 3, 2014, 6:19pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545 "2014-11-03T18:19:35Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![TealFawn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tealfawn/32/44845_2.png) [@TealFawn](https://discuss.elastic.co/u/TealFawn)\ **Post date:** [November 3, 2014, 6:19pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/1 "2014-11-03T18:19:35Z")
I'm running elasticsearch on Centos using the \*serivce elasticsearch start. \*The service happily stays running until it receives its first request causing it to die. _service elasticsearch status_ returns elasticsearch dead but pid file exists This same problem doesn't exist when I run elasticsearch with _./bin/elasticsearch_ With logging set to trace I see no stacktrace in the logs or any indication the service went down. Where else can I look to find why elasticsearch will not accept request running as a service? More info: Elasticsearch version:{ number: "1.3.4", build\_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45", build\_timestamp: "2014-09-30T09:07:17Z", build\_snapshot: false, lucene\_version: "4.9" CentOS 6.5 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alexandre\_Rafalovitc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexandre_rafalovitc/32/1124_2.png) [@Alexandre\_Rafalovitc](https://discuss.elastic.co/u/Alexandre_Rafalovitc)\ **Post date:** [November 3, 2014, 7:01pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/2 "2014-11-03T19:01:03Z")
Lack of permissions on filesystem access? Unexpected directory. Something like dtrace/truss/strace might be useful here. On 03/11/2014 1:19 pm, "Jef Statham" [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > I'm running elasticsearch on Centos using the \*serivce elasticsearch > start. \*The service happily stays running until it receives its first > request causing it to die. > _service elasticsearch status_ returns > > elasticsearch dead but pid file exists > > This same problem doesn't exist when I run elasticsearch with > _./bin/elasticsearch_ > > With logging set to trace I see no stacktrace in the logs or any > indication the service went down. > > Where else can I look to find why elasticsearch will not accept request > running as a service? > > More info: > Elasticsearch > version:{ > number: "1.3.4", > build\_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45", > build\_timestamp: "2014-09-30T09:07:17Z", > build\_snapshot: false, > lucene\_version: "4.9" > > CentOS 6.5 > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK\_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![TealFawn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tealfawn/32/44845_2.png) [@TealFawn](https://discuss.elastic.co/u/TealFawn)\ **Post date:** [November 3, 2014, 7:08pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/3 "2014-11-03T19:08:06Z")
Thanks for the strace suggestion, this is what my trace returns. I'm looking into what futex is now. [root@localhost elasticsearch]# sudo service elasticsearch start Starting elasticsearch: [OK] [root@localhost elasticsearch]# sudo service elasticsearch status elasticsearch (pid 16887) is running... [root@localhost elasticsearch]# strace -p 16887 \Process 16887 attached - interrupt to quit futex(0x7f97ed4279d0, FUTEX\_WAIT, 16889, NUL \ +++ killed by SIGKILL +++ [root@localhost elasticsearch]# sudo service elasticsearch start Starting elasticsearch: [OK] [root@localhost elasticsearch]# sudo service elasticsearch status elasticsearch (pid 16958) is running... [root@localhost elasticsearch]# strace -p 16958 Process 16958 attached - interrupt to quit futex(0x7f0a7e8089d0, FUTEX\_WAIT, 16960, NULL \ +++ killed by SIGKILL +++ _-Jef Statham_ Without vices there would be no virtues. It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring! On Mon, Nov 3, 2014 at 2:01 PM, Alexandre Rafalovitch [arafalov@gmail.com](mailto:arafalov@gmail.com) wrote: > Lack of permissions on filesystem access? Unexpected directory. > > Something like dtrace/truss/strace might be useful here. > On 03/11/2014 1:19 pm, "Jef Statham" [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > > > I'm running elasticsearch on Centos using the \*serivce elasticsearch > > start. \*The service happily stays running until it receives its first > > request causing it to die. > > _service elasticsearch status_ returns > > > > elasticsearch dead but pid file exists > > > > This same problem doesn't exist when I run elasticsearch with > > _./bin/elasticsearch_ > > > > With logging set to trace I see no stacktrace in the logs or any > > indication the service went down. > > > > Where else can I look to find why elasticsearch will not accept request > > running as a service? > > > > More info: > > Elasticsearch > > version:{ > > number: "1.3.4", > > build\_hash: "a70f3ccb52200f8f2c87e9c370c6597448eb3e45", > > build\_timestamp: "2014-09-30T09:07:17Z", > > build\_snapshot: false, > > lucene\_version: "4.9" > > > > CentOS 6.5 > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f71b44e6-540c-4213-a8bb-e1de5b2f6257%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK\_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK\_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G463EttK_d0uS4w-9fxAxter0gDx-M3ppHWhEuSB4L6g%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj8sWJPb2ATE8o%2Bpf36GNSF%3Dws7%3DkBo4pR%2BJmfjiXZ8rjw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj8sWJPb2ATE8o%2Bpf36GNSF%3Dws7%3DkBo4pR%2BJmfjiXZ8rjw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alexandre\_Rafalovitc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexandre_rafalovitc/32/1124_2.png) [@Alexandre\_Rafalovitc](https://discuss.elastic.co/u/Alexandre_Rafalovitc)\ **Post date:** [November 3, 2014, 8:43pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/4 "2014-11-03T20:43:04Z")
I was actually thinking more about using trace to monitor file system access (not just open, access as well). Regards, Alex. On 3 November 2014 14:08, Jef Statham [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > Thanks for the strace suggestion, this is what my trace returns. I'm looking > into what futex is now. Personal: [http://www.outerthoughts.com/](http://www.outerthoughts.com/) and @arafalov Solr resources and newsletter: [http://www.solr-start.com/](http://www.solr-start.com/) and @solrstart Solr popularizers community: [Sign Up | LinkedIn](https://www.linkedin.com/groups?gid=6713853) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![TealFawn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tealfawn/32/44845_2.png) [@TealFawn](https://discuss.elastic.co/u/TealFawn)\ **Post date:** [November 3, 2014, 9:16pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/5 "2014-11-03T21:16:21Z")
These debugging commands are neat I'm learning as we go, here's some file access stats: [root@localhost ~]# service elasticsearch start Starting elasticsearch: [OK] [root@localhost ~]# service elasticsearch status elasticsearch (pid 17129) is running... [root@localhost ~]# strace -f -e trace=file -p 17129 Process 17129 attached with 35 threads - interrupt to quit [pid 17163] lstat("/usr", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0 [pid 17163] lstat("/usr/share", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0 [pid 17163] lstat("/usr/share/elasticsearch", {st\_mode=S\_IFDIR|0777, st\_size=4096, ...}) = 0 [pid 17163] lstat("/usr/share/elasticsearch/lib", {st\_mode=S\_IFDIR|0777, st\_size=4096, ...}) = 0 [pid 17163] lstat("/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar", {st\_mode=S\_IFREG|0777, st\_size=2507813, ...}) = 0 [pid 17163] stat("/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.class", 0x7f1b301331c0) = -1 ENOENT (No such file or directory) [pid 17151] open("/usr/java/jdk1.8.0\_25/jre/lib/amd64/server/libjvm.so", O\_RDONLY) = 95 [pid 17151] open("/usr/java/jdk1.8.0\_25/jre/lib/amd64/server/libjvm.so", O\_RDONLY) = 95 ) = ? [pid 17172] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17172 leader 17129 [pid 17171] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17171 leader 17129 [pid 17170] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17170 leader 17129 [pid 17169] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17169 leader 17129 [pid 17168] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17168 leader 17129 [pid 17167] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17167 leader 17129 [pid 17166] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17166 leader 17129 [pid 17165] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17165 leader 17129 [pid 17162] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17162 leader 17129 [pid 17161] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17161 leader 17129 [pid 17160] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17160 leader 17129 [pid 17159] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17159 leader 17129 [pid 17158] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17158 leader 17129 [pid 17157] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17157 leader 17129 [pid 17156] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17156 leader 17129 [pid 17155] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17155 leader 17129 [pid 17154] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17154 leader 17129 [pid 17153] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17153 leader 17129 [pid 17152] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17152 leader 17129 [pid 17142] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17142 leader 17129 [pid 17164] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17164 leader 17129 [pid 17163] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17163 leader 17129 [pid 17141] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17141 leader 17129 [pid 17140] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17140 leader 17129 [pid 17139] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17139 leader 17129 [pid 17138] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17138 leader 17129 [pid 17137] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17137 leader 17129 [pid 17136] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17136 leader 17129 [pid 17135] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17135 leader 17129 [pid 17134] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17134 leader 17129 [pid 17133] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17133 leader 17129 [pid 17132] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17132 leader 17129 [pid 17131] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17131 leader 17129 [pid 17151] +++ killed by SIGKILL +++ PANIC: handle\_group\_exit: 17151 leader 17129 +++ killed by SIGKILL +++ _-Jef Statham_ Without vices there would be no virtues. It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring! On Mon, Nov 3, 2014 at 3:43 PM, Alexandre Rafalovitch [arafalov@gmail.com](mailto:arafalov@gmail.com) wrote: > I was actually thinking more about using trace to monitor file system > access (not just open, access as well). > > Regards, > Alex. > > On 3 November 2014 14:08, Jef Statham [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > > > Thanks for the strace suggestion, this is what my trace returns. I'm > > looking > > into what futex is now. > > Personal: [http://www.outerthoughts.com/](http://www.outerthoughts.com/) and @arafalov > Solr resources and newsletter: [http://www.solr-start.com/](http://www.solr-start.com/) and @solrstart > Solr popularizers community: [Sign Up | LinkedIn](https://www.linkedin.com/groups?gid=6713853) > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj\_52g5E%2BAocsT998jYrXHpVUcDEKv9qb0Vj%2Ba9qHxtvyQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj_52g5E%2BAocsT998jYrXHpVUcDEKv9qb0Vj%2Ba9qHxtvyQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![TealFawn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tealfawn/32/44845_2.png) [@TealFawn](https://discuss.elastic.co/u/TealFawn)\ **Post date:** [November 3, 2014, 9:27pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/6 "2014-11-03T21:27:54Z")
Updating my command, could there be an issue with the write lock on the events-production index? [root@localhost ~]# strace -f -e trace=file,read,write -p 17197 [pid 17231] stat("/var/lib/elasticsearch/elasticsearch/nodes/0/indices/events-production/2/index/write.lock", {st\_mode=S\_IFREG|0644, st\_size=0, ...}) = 0 [pid 17231] write(26, "[2014-11-03 16:18:31,244][DEBUG]"..., 138) = 138 [pid 17231] stat("/var/lib/elasticsearch/elasticsearch/nodes/0/indices/events-production/2/translog", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0 [pid 17231] write(26, "[2014-11-03 16:18:31,257][DEBUG]"..., 316) = 316 [pid 17231] open("/sys/devices/system/cpu/online", O\_RDONLY|O\_CLOEXEC) = 142 [pid 17231] read(142, "0\n", 8192) = 2 [pid 17231] write(26, "[2014-11-03 16:18:31,273][DEBUG]"..., 179) = 179 [pid 17231] write(26, "[2014-11-03 16:18:31,274][DEBUG]"..., 116) = 116 [pid 17231] write(26, "[2014-11-03 16:18:31,275][DEBUG]"..., 207) = 207 ) = ? [pid 17232] +++ killed by SIGKILL +++ _-Jef Statham_ Without vices there would be no virtues. It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring! On Mon, Nov 3, 2014 at 4:16 PM, Jef Statham [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > These debugging commands are neat I'm learning as we go, here's some file > access stats: > > [root@localhost ~]# service elasticsearch start > > Starting elasticsearch: [OK] > > [root@localhost ~]# service elasticsearch status > > elasticsearch (pid 17129) is running... > > [root@localhost ~]# strace -f -e trace=file -p 17129 > > Process 17129 attached with 35 threads - interrupt to quit > > [pid 17163] lstat("/usr", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) = 0 > > [pid 17163] lstat("/usr/share", {st\_mode=S\_IFDIR|0755, st\_size=4096, ...}) > = 0 > > [pid 17163] lstat("/usr/share/elasticsearch", {st\_mode=S\_IFDIR|0777, > st\_size=4096, ...}) = 0 > > [pid 17163] lstat("/usr/share/elasticsearch/lib", {st\_mode=S\_IFDIR|0777, > st\_size=4096, ...}) = 0 > > [pid 17163] lstat("/usr/share/elasticsearch/lib/lucene-core-4.9.1.jar", > {st\_mode=S\_IFREG|0777, st\_size=2507813, ...}) = 0 > > [pid 17163] > stat("/org/apache/lucene/codecs/lucene42/Lucene42DocValuesFormat.class", > 0x7f1b301331c0) = -1 ENOENT (No such file or directory) > > [pid 17151] open("/usr/java/jdk1.8.0\_25/jre/lib/amd64/server/libjvm.so", > O\_RDONLY) = 95 > > [pid 17151] open("/usr/java/jdk1.8.0\_25/jre/lib/amd64/server/libjvm.so", > O\_RDONLY) = 95 > > ) = ? > > [pid 17172] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17172 leader 17129 > > [pid 17171] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17171 leader 17129 > > [pid 17170] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17170 leader 17129 > > [pid 17169] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17169 leader 17129 > > [pid 17168] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17168 leader 17129 > > [pid 17167] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17167 leader 17129 > > [pid 17166] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17166 leader 17129 > > [pid 17165] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17165 leader 17129 > > [pid 17162] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17162 leader 17129 > > [pid 17161] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17161 leader 17129 > > [pid 17160] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17160 leader 17129 > > [pid 17159] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17159 leader 17129 > > [pid 17158] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17158 leader 17129 > > [pid 17157] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17157 leader 17129 > > [pid 17156] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17156 leader 17129 > > [pid 17155] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17155 leader 17129 > > [pid 17154] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17154 leader 17129 > > [pid 17153] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17153 leader 17129 > > [pid 17152] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17152 leader 17129 > > [pid 17142] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17142 leader 17129 > > [pid 17164] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17164 leader 17129 > > [pid 17163] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17163 leader 17129 > > [pid 17141] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17141 leader 17129 > > [pid 17140] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17140 leader 17129 > > [pid 17139] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17139 leader 17129 > > [pid 17138] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17138 leader 17129 > > [pid 17137] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17137 leader 17129 > > [pid 17136] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17136 leader 17129 > > [pid 17135] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17135 leader 17129 > > [pid 17134] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17134 leader 17129 > > [pid 17133] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17133 leader 17129 > > [pid 17132] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17132 leader 17129 > > [pid 17131] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17131 leader 17129 > > [pid 17151] +++ killed by SIGKILL +++ > > PANIC: handle\_group\_exit: 17151 leader 17129 > > +++ killed by SIGKILL +++ > > _-Jef Statham_ > > Without vices there would be no virtues. > It’s a magical world, Hobbes, ol’ buddy…Let’s go exploring! > > On Mon, Nov 3, 2014 at 3:43 PM, Alexandre Rafalovitch [arafalov@gmail.com](mailto:arafalov@gmail.com) > wrote: > > > I was actually thinking more about using trace to monitor file system > > access (not just open, access as well). > > > > Regards, > > Alex. > > > > On 3 November 2014 14:08, Jef Statham [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > > > > > Thanks for the strace suggestion, this is what my trace returns. I'm > > > looking > > > into what futex is now. > > > > Personal: [http://www.outerthoughts.com/](http://www.outerthoughts.com/) and @arafalov > > Solr resources and newsletter: [http://www.solr-start.com/](http://www.solr-start.com/) and @solrstart > > Solr popularizers community: [Sign Up | LinkedIn](https://www.linkedin.com/groups?gid=6713853) > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe](https://groups.google.com/d/topic/elasticsearch/2tIe5i2b-TA/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-Gu8nQf%3DgfxvWvkSwX1vrN50-zkkauA8iPnWb4n8zfPcA%40mail.gmail.com) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj9a7LHywxVy6HQAgxLkG%2BKm\_e2X1tVjpnRiMKtHsun9og%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CABj%3DZj9a7LHywxVy6HQAgxLkG%2BKm_e2X1tVjpnRiMKtHsun9og%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alexandre\_Rafalovitc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexandre_rafalovitc/32/1124_2.png) [@Alexandre\_Rafalovitc](https://discuss.elastic.co/u/Alexandre_Rafalovitc)\ **Post date:** [November 3, 2014, 9:33pm UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/7 "2014-11-03T21:33:15Z")
These look like debug messages. Do you know where they go and what they say. Might be where the clue is. Otherwise, I am not sure. The usual debugging method is to try comparing failing instance with a successful one (on file access sequence). You could try that with a fresh local copy of the ES and see. Regards, Alex. On 3 November 2014 16:27, Jef Statham [jef.statham@gmail.com](mailto:jef.statham@gmail.com) wrote: > [pid 17231] write(26, "[2014-11-03 16:18:31,273][DEBUG]"..., 179) = 179 > [pid 17231] write(26, "[2014-11-03 16:18:31,274][DEBUG]"..., 116) = 116 > [pid 17231] write(26, "[2014-11-03 16:18:31,275][DEBUG]"..., 207) = 207 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G-h%2BVbvaZJU53pP6Z8abh8OuLb1%3Dqh1Qz3ORXXFTiDSA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEFAe-G-h%2BVbvaZJU53pP6Z8abh8OuLb1%3Dqh1Qz3ORXXFTiDSA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:52am UTC](https://discuss.elastic.co/t/elasticsearch-run-as-a-service-dies-on-first-request/20545/8 "2017-07-06T00:52:24Z")