Could not index event to Elasticsearch: Maximum shards open

I am creating indexes on Elasticsearch daily from metricbeat, auditbeat, winlogbeat, and other logs. I have run into the issue of getting an error on the logstash saying "Validation Failed: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open".

Here is a more detailed log of the error:

Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e327a87>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5caece52>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1fc5b16f>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1901cbeb>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e354641>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5624c1f3>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x78945d6f>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}

On Kibana:

When I run the command:GET _cat/health?v

It gives me the following output:

epoch      timestamp cluster              status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1566505132 20:18:52 dev-cluster           yellow          1         1    520 520    0    0      511             9               1.1s                 50.4%

Currently, I have 504 indices open.

If I close the indexes, it helps me to recover from this issue temporally. Until more indexes are created and reach the limit.

Can you help me fix this issue? Thank you!

You only have 1 data node. So you can only have 1 shard and no replica for such cluster to be green, which is not good.
I believe all the unassigned shards are because there are no additional data nodes to assign to. It should've been a day one issue.
You need at least 2 data nodes to allow 1 replica so primary & replica shards are on 2 different nodes.

You are reaching the limit 'cluster.max_shards_per_node'. Add more data node or reduce your shards of cluster.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.