Unable to bulk upload the stats payload to the local cluster Kibana 7.3

[Mohamad-Sobhie]

Hey there,

Kibana 7.3

I recieve this message when attempting to bulk :

"tags":["warning","monitoring","kibana-monitoring"],"pid":1,"message":"Unable to bulk upload the stats payload to the local cluster"}

Note that I am testing my elastic stack by generating lots of documents every hour.

my JVM Heap

my indexing rate and searching rate

---

some logs of my elasticsearch node

"Caused by: org.elasticsearch.common.ValidationException: Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;",
"at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.checkShardLimit(MetaDataCreateIndexService.java:657) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.checkShardLimit(MetaDataCreateIndexService.java:613) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:446) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310) ~[elasticsearch-7.3.0.jar:7.3.0]",
"at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210) ~[elasticsearch-7.3.0.jar:7.3.0]",
"... 9 more"] }

---

[Mike_Place]

Hi @Mohamad-Sobhie

Could you please describe how you are "generating lots of documents"? Just by guessing from the error you're seeing, I am wondering if you are creating many indices and, in turn, many primary shards.

If you go to the "Overview" page in Stack Monitoring for your Elasticsearch cluster, you should see toward the top a bar with metrics, which shows "Total Shards", "Unassigned Shards", and "Total Documents". What are those values?

Best,

-Mike

[Mohamad-Sobhie]

yes I am creating indices (not sure how many indices is many). Here is my overview

[Mike_Place]

You should aim to keep the number of shards per node as low as possible for the expected load.

There is a good introduction to designing for scale which can be found here: https://www.elastic.co/guide/en/elasticsearch/guide/2.x/scale.html

I might also recommend this webinar on quantitative sizing which should help you to better understand how to test and size an Elasticsearch cluster: https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing