[Tribe] ghost indices on tribes

Elastic Stack Elasticsearch
1

Hello,

Using tribe nodes on ES 1.7.5, we experienced some unexpected behavior.
How to reproduce:

# Create 2 clusters + 1 tribe
bin/elasticsearch --cluster.name ldn
bin/elasticsearch --cluster.name hk
bin/elasticsearch --tribe.ldn.cluster.name ldn --tribe.hk.cluster.name hk

# On ldn
curl -XPUT localhost:9200/conflict_index

# On hk
curl -XPUT localhost:9201/conflict_index

# conflict_index is visible on tribe node, from tribe ldn
curl localhost:9202/conflict_index?pretty

# Delete index on ldn tribe
curl -XDELETE localhost:9200/conflict_index

When getting index on tribe (curl localhost:9202/conflict_index?pretty), we get an IndexMissingException. The tribe node does not see any index, when there is one on tribe 2.

Is this indeed a bug?

Regards,

Jean

2

The way the tribe node works is that it listens to cluster state updates from all tribes and updates its own internal cluster state, which is a merge of all the information it ever received. The problem with same index names on multiple tribes, is that the tribe node can only know about one. So either you set the on_conflict setting to instruct the tribe node on which one to keep, or it will choose either one of the two.

The problem is that if you then delete that one index that the tribe node sees, the deletion will be propagated to the tribe node. At that point the tribe node doesn't know about the other index with same name on the other tribe, as that cluster state update was received and previously rejected . That index will not come back then, although it would be seen if it was re-created at that time.

Have a look at the docs to know how to set up the on_conflict setting if needed: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-tribe.html .