Bad recovery after cluster restart

Elastic Stack Elasticsearch
1

Hi,

We are working on ES 0.17.6 with 2 servers and 4 ES nodes.
After a restart on cluster, we have lost all indices.
We can see them in work folders but the cluster state is green (cluster
health: green (4, 0)). but nothing happens concerning the indices.

the gist with configuration and log

any ideas how to force recovery ?

Thanks you.

2

  • I am a bit confused about your setup. You are using the shared fs
    gateway, is the path you configure it with (the work directory of ES???)
    mounted on both servers and both servers see the same file system?

  • Why are you using the shared fs gateway and not the local gateway?

  • Why are you starting two nodes on a single server?

On Tue, Nov 22, 2011 at 2:16 PM, Ludovic superglu07@gmail.com wrote:

Hi,

We are working on ES 0.17.6 with 2 servers and 4 ES nodes.
After a restart on cluster, we have lost all indices.
We can see them in work folders but the cluster state is green (cluster
health: green (4, 0)). but nothing happens concerning the indices.

the gist with configuration and log

bad recovery after cluster restart · GitHub

any ideas how to force recovery ?

Thanks you.

3

Hi Shay

  • I am a bit confused about your setup. You are using the shared fs
    gateway, is the path you configure it with (the work directory of ES???)
    mounted on both servers and both servers see the same file system?

each nodes has its own "work" directory.

  • Why are you using the shared fs gateway and not the local gateway?

We have only 2 servers with not enough disk memory, so we can't do anything
else then using a disk storage bay. that is why we used this shared fs
gateway.

  • Why are you starting two nodes on a single server?

The indices we want to use have 1 replica. We want to test load balancing
for 4 nodes and only 2 servers. We understand, but we might be wrong, that
4 nodes will be better than 2.

best regards

4

On Tue, Nov 22, 2011 at 4:06 PM, Ludovic superglu07@gmail.com wrote:

Hi Shay

  • I am a bit confused about your setup. You are using the shared fs
    gateway, is the path you configure it with (the work directory of ES???)
    mounted on both servers and both servers see the same file system?

each nodes has its own "work" directory.

then this does not work. The shared fs gateway requires the same mount to
be visible from all servers. But, you don't really need it, use the
default, recommended local gateway.

That explains, btw, why you "lost" your data. the master node stores in the
presumed shared gateway location the fact that indices were created. But,
because your location is not shared, when you started and a master node was
elected on another server, it will go to its own "shared" fs location, and
there won't be any meta data there.

  • Why are you using the shared fs gateway and not the local gateway?

We have only 2 servers with not enough disk memory, so we can't do
anything else then using a disk storage bay. that is why we used this
shared fs gateway.

The local gateway also works using the "file system":
Elasticsearch Platform — Find real-time answers at scale | Elastic. I
suggest you use it. You will need to reindex the data to move from shared
fs gateway to local gateway.

  • Why are you starting two nodes on a single server?

The indices we want to use have 1 replica. We want to test load balancing
for 4 nodes and only 2 servers. We understand, but we might be wrong, that
4 nodes will be better than 2.

99% change you are wrong, use one node per machine.

best regards