Total Shards Differs (dev console vs. script output)

Hi lovely Elastic folks,

I have a script that rolls over a few data streams and my cluster quickly became oversharded.

The rollover requests are now failing with:
elasticsearch.exceptions.RequestError: RequestError(400, 'validation_exception', 'Validation Failed: 1: this action would add [2] shards, but this cluster currently has [999]/[1000] maximum normal shards open;')

A dev console request for shards on the cluster shows only 508:
GET _cluster/stats?filter_path=indices.shards.total

{
  "indices" : {
    "shards" : {
      "total" : 508
    }
  }
}

Anyone know why there's a discrepancy?
Bonus questions: Does every new backing index on a data stream have to have it's own shards? And why are there 2 shards per rollover, instead of 1?

Happy to be a part of a cool community

I'd say the cluster stats is primary only, where as the limit is primary and replica.

Ah nice, that makes sense as the query gets around half the max. Warkolm thanks for the help yet again!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.