Running more than one ES instance on the same server while indexing data


(Frederic) #1

Hey guys, hope you can help me with this.

By mistake, two instances of ES were launched (at the same 9200 port)
on al least 4 servers of a 8 nodes cluster. That happened while
multiple processes were indexing docs.

After shutting down these extra instances and restarting the cluster,
the state was in RED as 2 shards were unassigned. Today I restarted
all nodes one by one and the cluster health got Green.

From my tests on localhost, new ES instances running on the same
machine form a cluster like any other node and reallocate shards among
them, is this correct?

If so, I understand that shutting down ES instances would act as if a
node was stopped, regardless if some instances are running on the same
server, correct?

If not... is there any chance that information sotred previously to
the problem could be lost/corrupted because sharding reallocation
between these multiple instances of ES?

Thanks a lot for this great product,


(Karussell) #2

After shutting down these extra instances and restarting the cluster,
the state was in RED as 2 shards were unassigned.

How did you shut them down? one by one until cluster reached green?

E.g. if you shutdown two instances which have two replicas of a shard
(and you only have two) this shard cannot be reassigned ...

From my tests on localhost, new ES instances running on the same
machine form a cluster like any other node and reallocate shards among them, is this correct?

yes, but not only on the same machine ... you can avoid that via the
cluster name

If so, I understand that shutting down ES instances would act as if a
node was stopped, regardless if some instances are running on the same server, correct?

what do you mean here?

If not... is there any chance that information sotred previously to the problem could be lost/corrupted because sharding
reallocation between these multiple instances of ES?

IMO no

Regards,
Peter.

On Nov 18, 9:03 pm, Frederic focampo...@gmail.com wrote:

Hey guys, hope you can help me with this.

By mistake, two instances of ES were launched (at the same 9200 port)
on al least 4 servers of a 8 nodes cluster. That happened while
multiple processes were indexing docs.

After shutting down these extra instances and restarting the cluster,
the state was in RED as 2 shards were unassigned. Today I restarted
all nodes one by one and the cluster health got Green.

From my tests on localhost, new ES instances running on the same
machine form a cluster like any other node and reallocate shards among
them, is this correct?

If so, I understand that shutting down ES instances would act as if a
node was stopped, regardless if some instances are running on the same
server, correct?

If not... is there any chance that information sotred previously to
the problem could be lost/corrupted because sharding reallocation
between these multiple instances of ES?

Thanks a lot for this great product,


(Frederic) #3

Hi Peter,

Thanks for your help, and patiente too (my english is far from good :slight_smile:

How did you shut them down? one by one until cluster reached green?
E.g. if you shutdowntwoinstanceswhich havetworeplicas of a shard
(and you only havetwo) this shard cannot be reassigned ...

Actually I shutted them all down, one by one (i.e. kill $PID), and
restarted again one by one until status reached green.

yes, but not only on the same machine ... you can avoid that via the
cluster name

yap, that makes sense

If so, I understand that shutting down ES instances would act as if a
node was stopped, regardless if some instances are running on the same server, correct?

what do you mean here?

sorry if it is not very clear. I just meant that if I have two ES
instances running on the same machine, both part of the same cluster,
and I shut down one of them, that would be the same as if I shut down
an ES running on a different node. From your answers I think yes.

If not... is there any chance that information sotred previously to the problem could be lost/corrupted because sharding
reallocation between these multiple instances of ES?

IMO no

same for me and that's really great :¬)

Thanks again Peter,

Frederic


(Shay Banon) #4

On Fri, Nov 18, 2011 at 10:03 PM, Frederic focampo.br@gmail.com wrote:

Hey guys, hope you can help me with this.

By mistake, two instances of ES were launched (at the same 9200 port)
on al least 4 servers of a 8 nodes cluster. That happened while
multiple processes were indexing docs.

After shutting down these extra instances and restarting the cluster,
the state was in RED as 2 shards were unassigned. Today I restarted
all nodes one by one and the cluster health got Green.

From my tests on localhost, new ES instances running on the same
machine form a cluster like any other node and reallocate shards among
them, is this correct?

Yes. Though you can disable allowing to run more than one node on a machine
by setting node.max_local_storage_nodes setting to 1.

If so, I understand that shutting down ES instances would act as if a
node was stopped, regardless if some instances are running on the same
server, correct?

Yes. So, if you brought down the nodes one by one, and waited for green
state between each shutdown, then all shards would end up migrated to the
remaining nodes.

If not... is there any chance that information sotred previously to
the problem could be lost/corrupted because sharding reallocation
between these multiple instances of ES?

Thanks a lot for this great product,


(system) #5