Indices won't close

Elastic Stack Elasticsearch
1

Hello, first post, still learning.

Having an issue closing "random" indices, after removing replica shards and using shard allocation awareness. ES 6.8.3, although indices may have been created in ES 5 (I know).

  1. Remove Replicas (set 0)
  2. Add "require" to index for shard allocation awareness.
  3. Let ES move the index to the node, attempt to close in the index.

Using POST //_close

Getting this error message in debug:
"verification of shards before closing {} failed [{}]"

There is literally no further information in the log.

From what I am able to see, after removing the replica shards (setting replicas to 0), allocating the shards to a different node using allocation awareness, "random" indices won't close.

I am unable to find any other topics related to this anywhere.

If I run:

/_cluster/state/metadata,routing_table/

I can see that the removed replicas are still referenced in the index metadata under "in_sync_allocations", but not in the routing table.

I can "resolve" the issue by moving the index back to the original node, adding replicas back, closing the index (it will close after adding replicas back and moving back), removing the replicas, setting require, opening the index so it can be assigned/moved to the required node, once moved, the index will now close.

If I look at the cluster state for the index, the replica shards are no longer referenced in the metadata for the index.

Is this an order of operations, or doing operations too fast? I'm not clear why removing the replicas a second time, finally removes the replicas from the "in_sync_allocations". Once this done, indices will close/open as desired on any assigned node.

Any input appreciated!

3

verification of shards before closing {} failed [{}]

Can you provide the full message here, in particular the part after failed?

Can you reproduce this on 6.8.latest?

4

Sorry for the delay here. Thank you for the response. Still working on this as I still don't understand the root cause.

Here is an example of the log, edited to remove names, but that is the only thing.

The failed[{}] is empty on this each time

[DEBUG][o.e.c.s.MasterService ] [hostname] processing [add-block-index-to-close [[index1/aAQjNWhEQHyx0OUZBOm1tQ], [index2/f9jIE1dLT7m0pMLBgOQoHg], [index3/Pmxrv-6yS3ugGjNwG19sCQ]]]: took [37ms] done publishing updated cluster state (version: 135871, uuid: Re2CjyxHTDesJCDk-TzWVQ)
[DEBUG][o.e.c.s.MasterService ] [hostname] processing [close-indices]: execute
[DEBUG][o.e.c.m.MetaDataIndexStateService] [hostname] verification of shards before closing [indexname1/aAQjNWhEQHyx0OUZBOm1tQ] failed
[DEBUG][o.e.c.m.MetaDataIndexStateService] [hostname] verification of shards before closing [index2/f9jIE1dLT7m0pMLBgOQoHg] failed
[DEBUG][o.e.c.m.MetaDataIndexStateService] [hostname] verification of shards before closing [index3/Pmxrv-6yS3ugGjNwG19sCQ] failed
[INFO ][o.e.c.m.MetaDataIndexStateService] [hostname] completed closing of indices
[DEBUG][o.e.c.s.MasterService ] [hostname] cluster state updated, version [135872], source [close-indices]
[DEBUG][o.e.c.s.MasterService ] [hostname] publishing cluster state version [135872]

The response is Acknowledged false, and in Debug there is literally no other information.

I have started reviewing:

But my java is not that strong, and I can't determine what is actually failing. Any pointers would be great.

Thanks!

5

The error message was unfortunately only improved here:

As I said earlier, have you tried upgrading to 6.8.latest to see if the issue still exists there?

This might be a bug that's already fixed.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.