Index not found right after creation

Elastic Stack Elasticsearch
1

Elasticsearch version: 6.4.3

Our cluster has a large number of indexes (~300-400) and we occasionally create about 20 indexes at once. Most of the time everything works perfectly. However, sometimes we are experiencing the following sequence of events:

  1. Index my_index is created
  2. HEAD /my_index returns 200
  3. Any attempt to index an object to the index fails with a 404 (as does a call like GET /my_index/_settings

We've implemented some exponential backoff logic to wait some amount of time between steps 2 and 3 and it seems to help mitigate the problem, but doesn't eliminate the problem completely. We also make some cluster health calls with wait_for_status=green to really make sure the cluster is in good shape before proceeding.

  • Is this a known issue with ES that happens when there are large numbers of indexes?
  • Is there any setting or call we can make to help avoid this issue? I find it particularly strange that the HEAD call returns 200 but subsequent calls fail.

Thanks for any help!

2

How many shards do these indices have? How large is your cluster?

3

2 shards per index, running on two nodes, each 16GB. Memory pressure doesn't really get out of control and disk usage is low too.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.