Total Shards Differs (dev console vs. script output)

Elastic Stack Elasticsearch
1

Hi lovely Elastic folks,

I have a script that rolls over a few data streams and my cluster quickly became oversharded.

The rollover requests are now failing with:
elasticsearch.exceptions.RequestError: RequestError(400, 'validation_exception', 'Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open;')

A dev console request for shards on the cluster shows only 508:
GET _cluster/stats?filter_path=indices.shards.total

{
  "indices" : {
    "shards" : {
      "total" : 508
    }
  }
}

Anyone know why there's a discrepancy?
Bonus questions: Does every new backing index on a data stream have to have it's own shards? And why are there 2 shards per rollover, instead of 1?

Happy to be a part of a cool community

2

I'd say the cluster stats is primary only, where as the limit is primary and replica.

3

Ah nice, that makes sense as the query gets around half the max. Warkolm thanks for the help yet again!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.