Elasticsearch not starting up - ES 1.7/2.1 RHEL 6.4

Elastic Stack Elasticsearch
1

This is rather strange. It was working fine. But now it just doesn't get started and is stuck at
[2015-12-15 14:57:25,021][INFO ][node ] [Vertigo] initializing ...
[2015-12-15 14:57:25,079][INFO ][plugins ] [Vertigo] loaded [], sites []

Then I tried getting ES 2.1. But it just doesn't throw any logs at all and is completely stuck. I am unable to figure out what's wrong. Any ideas?

2

Figured out that "df" was stuck due to stale NFS mount causing this unwarranted behaviour. I guess ES must be trying to figure out the high/low disk watermarks prior to start-up.

3

Hello Satish,

There are other issues around using Elasticsearch on NFS as well - it is not recommended to use Elasticserach on NFS.

Cheers,
-Robin-

4

No no Robin, I am not using NFS for storing indices but the machine has NFS mounts which are not being used by ES at all. However I suspect it tries to do a "df" kind of operation where it got stuck!

5

Hi, can you post the full stack trace for the hung thread?

"kill -QUIT pid" should show it, where pid is the process ID of the Elasticsearch java process.

Thanks.

6

As a side node, please use bg,intr,soft mount option in /etc/fstab or when mounting NFS manually.

ES is indeed calling JVM file system services that list all mounted file systems, which is similar to df. By default, mounted NFS volumes, once they got stale, wait indefinitely for I/O.

When mounted with soft option, I/O requests on stale NFS mounts will timeout.

bg spins off a background retry task at mount time so NFS mounts do not block the OS to boot.

intr makes the programs who perform I/O requests on a stale NFS mount interruptible, e.g. by kill

1 Like
7

Excellent, thanks Jorg!