# What are the specifications of your shard configuration or equipment? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 16, 2021, 2:49pm UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155 "2021-06-16T14:49:47Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [June 16, 2021, 2:49pm UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/1 "2021-06-16T14:49:47Z")
I'm using Filebeat, Logstash and ElasticSearch to collect logs. Today, I encountered the following error. ```auto # tail /var/log/logstash/logstash-plain.log [2021-06-16T10:10:48,509][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,509][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,509][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,509][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,510][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,510][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,510][WARN][logstash.outputs.elasticsearch][main][c24e7e41b335e6af9290ccdd216db539ac55eeb30dbb4362e3e58cc79b91102b] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-cas-server-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-cas-server-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,510][WARN][logstash.outputs.elasticsearch][main][3fa00590784976059791f8c2c84cc91e565e159f5722752cfffdc4a041e5b050] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"sro-api-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"sro-api-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,512][WARN][logstash.outputs.elasticsearch][main][f0a309d0296c0cfaa8ca0029d1526d531aee25749bcb74718363b7d158aa718e] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-api-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-api-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} [2021-06-16T10:10:48,512][WARN][logstash.outputs.elasticsearch][main][f0a309d0296c0cfaa8ca0029d1526d531aee25749bcb74718363b7d158aa718e] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"madam-api-2021.06.16", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"madam-api-2021.06.16", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1076]/[1000] maximum shards open;"}}}} ``` This means that the shard cannot be indexed because the maximum number of shards exceeds the limit. I have taken steps to extend the maximum number from 1000 to 2000, referring to the following article. > > > hi @jm404 @manuasir @Zenidd > i can't use my kibana for management and monitorin…g. I have this message on discover page on kibana: > > \> Settings. 2011 - Could not save data in elasticsearch due to \[validation\_exception\] Validation Failed: 1: this action would add \[2\] total shards, but this cluster currently has \[1000\]/\[1000\] maximum shards open; (/elastic/api) > > I dont have cluster I have 1 wazuh server. > I did these : > \- thread\_pool.search.queue\_size: 10000 in /etc/elasticsearch/elasticsearch.yml but it wasn't solution and I deleted it > \- Xms8g & Xmx8g in jvm.options are set. my RAM is 16G > > the result of curl http://localhost:9200/\_cluster/health?pretty: > { > "cluster\_name" : "elasticsearch", > "status" : "yellow", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "number\_of\_data\_nodes" : 1, > "active\_primary\_shards" : 987, > "active\_shards" : 987, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 13, > "delayed\_unassigned\_shards" : 0, > "number\_of\_pending\_tasks" : 0, > "number\_of\_in\_flight\_fetch" : 0, > "task\_max\_waiting\_in\_queue\_millis" : 0, > "active\_shards\_percent\_as\_number" : 98.7 > } > my elasticsearch version is 7.3 > kibana 7.3 > I did all of things in this link" > \[https://documentation.wazuh.com/3.9/installation-guide/installing-elastic-stack/elastic\_tuning.html\](url) > but I still have error message 🤒 > can anyone help me? I can't check my agents :(((((( However, I think this is a temporary measure. Upon further investigation, I found the following article. > **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/jp/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** I intend to collect 20 different logs each day, keeping a date for each type, and keeping the logs for 60 days. I am preparing a 4TB disk to hold the 20 indexes for 60 days. (We think we can fit in this) At this point, what settings do I need to make to suppress the above error? Or what are the specs of the equipment needed for this? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2021, 2:16am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/2 "2021-06-17T02:16:13Z")
> [@its-ogawa](#): > > I intend to collect 20 different logs each day Are they all the same format? If not then grouping them into similar indices makes sense to limit mapping explosions like you are seeing. As for your other questions, the current best approach is to use [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) to optimise sharding. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [June 17, 2021, 2:28am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/3 "2021-06-17T02:28:33Z")
To be more precise, there are five different formats that are the same, and we collect logs from four different services. One service has 2 to 10 servers outputting logs daily. Is it unreasonable to do this with a single ElasticSearch? We use `curator` for the index lifecycle, and we delete indexes older than 60 days. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2021, 2:29am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/4 "2021-06-17T02:29:50Z")
> [@its-ogawa](#): > > there are five different formats that are the same Not sure I follow how they can be different and the same? ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [June 17, 2021, 2:38am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/5 "2021-06-17T02:38:43Z")
> [@warkolm](#): > > Not sure I follow how they can be different and the same? Do you mean that it would be easier to make suggestions if there was a specific log format? ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [June 30, 2021, 2:10am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/6 "2021-06-30T02:10:57Z")
I worked around the error tentatively by running the following command. ```auto $ curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "2000" } ``` What I want to know is the upper limit (or upper bound?) of the number of shards. If it is due to machine specs, etc., I would like to know the connection. The reason I want to know this is so that I can consider what spec server will be able to hold up to the number of logs I want to store in ElasticSearch (i.e., the factor that increases the number of shards), and whether I need to expand the server. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [July 2, 2021, 3:26am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/7 "2021-07-02T03:26:02Z")
Would it be better to be more specific? There are a few questions. - If I don't create replicas and just create one primary shard, can I assume that the number of shards equals the number of indexes? - Is it the disk size that you are referring to here? Or is it the memory size? - Can we raise the upper limit on the number of shards if we have enough disk size? - If we keep the size per shard at 40GB, is it correct that we can set the limit based on the number of shards as long as the total shard size does not exceed the disk space? - Does it count the number of shards even when no replicas are created? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 2, 2021, 5:41am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/8 "2021-07-02T05:41:12Z")
> [@its-ogawa](#): > > If I don't create replicas and just create one primary shard, can I assume that the number of shards equals the number of indexes? If each index has one primary shard and there are no replicas configured that would be correct. > [@its-ogawa](#): > > Is it the disk size that you are referring to here? Or is it the memory size? The blog post refers to Java heap size. Each shard has some over head in terms of heap usage and also contributes to the size of the cluster state as information about the shard(s) and mappinmgs need to be stored there. For large clusters the size of the cluster state can be a limiting factor even if heap usage looks OK as updates to the clkuster state may get slow and it changes can be slow to propagate to other nodes. > [@its-ogawa](#): > > Can we raise the upper limit on the number of shards if we have enough disk size? As described earlier the amount of data a node can handle is often limited by the heap size and not always disk space. > [@its-ogawa](#): > > If we keep the size per shard at 40GB, is it correct that we can set the limit based on the number of shards as long as the total shard size does not exceed the disk space? The blog post outlines a guideline around the limit of shards per GB heap but as memory usage per shard will depend on shard size as well as mappings there is no guarantee a node can handle a certain number of large shards at this limit. > [@its-ogawa](#): > > Does it count the number of shards even when no replicas are created? Yes, primary and replica shards both count towards the limit. > [@its-ogawa](#): > > In other words, if you have 1 primary shard and 0 replicas, is the shard count 1 or 2? That is a single shard. I would provide the following recommendations: - If you are going to run a single node cluster (it sounds like this might be what you are planning) you will never have any replica shards as Elasticsearch will not allocate replicas to the same node where the primary resides. This will leave you without high availability which may be OK for your use case. In this case I think you can increase the limit to 2000 as long as you have enough heap to handle the data as 2000 indices in the cluster state should be fine. If you were planning a larger cluster with more data nodes I would generally avoid changing the limit. - In order to reduce heap usage and increase the amount of data your node can handle I would recommend forcemerging indices down to a single segment and make them read-only once they are no longer written to as this can reduce heap usage significantly. You should be able to automate this using ILM by specifying a hot and warm phase without relocation of shards. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [July 2, 2021, 6:13am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/9 "2021-07-02T06:13:23Z")
Thank you for your answer!!! I'm slowly getting my head around this. In particular, thank you for your recommendations. First of all, as you have suggested, I will not be allocating replicas in a single node cluster. The reason for this is that in my case, it is the server logs that I am collecting, and the logs themselves are stored on each server, so I don't think replicas are necessary on elasticsearch. In order to keep the daily logs for 60 days, the other day the number of shards exceeded 1000 and logstash stopped. As a first aid, I changed the upper limit of the number of shards to 2000, which turned out to be a mistake. I am relieved. Your second suggestion is very interesting!!! Could you please elaborate on how to do that? I use elastic's curator to manage my ILM. By indexing daily, it is very easy to configure curator to delete indexes older than 60 days. However, the heap is always hovering near the memory limit (16GB) and we want to control this. We can make it read-only, so it doesn't need to be written to except for the index that is collecting the logs today. (I would like to know how to do that). Also, if I consolidate the indexes into one segment, is it easy to delete the indexes by date? (I would like to know how to do that if possible). ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 2, 2021, 6:25am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/10 "2021-07-02T06:25:05Z")
If you only have 16GB heap that will limit the amount of data you can hold on the node and you should not increase the shard limit in that case. I assumed you were using the maximum heap around 30GB. In a single node cluster you can never allocate replica shards so that is not a concern. Curator has even more options and features than ILM so you can do everything I described using Curator. I have not configured Curator in a while so will however not be able to help with that. > [@its-ogawa](#): > > Also, if I consolidate the indexes into one segment, is it easy to delete the indexes by date? (I would like to know how to do that if possible). Forcemerging into a single segment just optimised the shards and this does not affect how you manage retention. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [July 2, 2021, 6:33am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/11 "2021-07-02T06:33:27Z")
> [@Christian\_Dahlqvist](#): > > If you only have 16GB heap that will limit the amount of data you can hold on the node and you should not increase the shard limit in that case. I assumed you were using the maximum heap around 30GB. Hmmm... if I only have a 16GB heap, should I leave the maximum number of shards at 1000? If so, does that mean I need to increase the memory to 30GB or have another server? > [@Christian\_Dahlqvist](#): > > Curator has even more options and features than ILM so you can do everything I described using Curator. I have not configured Curator in a while so will however not be able to help with that. I see. curator is a very nice tool. I'll look into it more. > [@Christian\_Dahlqvist](#): > > Forcemerging into a single segment just optimised the shards and this does not affect how you manage retention. So, I don't need to change the curator settings, right? I'm relieved. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 2, 2021, 6:36am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/12 "2021-07-02T06:36:01Z")
> [@its-ogawa](#): > > Hmmm... if I only have a 16GB heap, should I leave the maximum number of shards at 1000? > If so, does that mean I need to increase the memory to 30GB or have another server? Yes, I think that is what I would recommend. > [@its-ogawa](#): > > So, I don't need to change the curator settings, right? I'm relieved. You will need to add additional actions to perform the forcemerge and make the index read only. ---
**Author:** ![its-ogawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/its-ogawa/32/120829_2.png) [@its-ogawa](https://discuss.elastic.co/u/its-ogawa)\ **Post date:** [July 2, 2021, 6:46am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/13 "2021-07-02T06:46:45Z")
> [@Christian\_Dahlqvist](#): > > > [@its-ogawa](#): > > > > Hmmm... if I only have a 16GB heap, should I leave the maximum number of shards at 1000? > > > > > If so, does that mean I need to increase the memory to 30GB or have another server? > > Yes, I think that is what I would recommend. I understand. I'll try to add more memory or server. (Thank you very much. I've been wanting this advice for a long time!) > [@Christian\_Dahlqvist](#): > > > [@its-ogawa](#): > > > > So, I don't need to change the curator settings, right? I'm relieved. > > You will need to add additional actions to perform the forcemerge and make the index read only. Thanks for the keywords. Does the forcemerge you refer to refer to the one on the following page? > **[Forcemerge | Curator Reference \[8.0\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/forcemerge.html#forcemerge)** If so, I'll read it and add it to the curator's actions. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 30, 2021, 6:46am UTC](https://discuss.elastic.co/t/what-are-the-specifications-of-your-shard-configuration-or-equipment/276155/14 "2021-07-30T06:46:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.