I am creating indexes on Elasticsearch daily from metricbeat, auditbeat, winlogbeat, and other logs. I have run into the issue of getting an error on the logstash saying "Validation Failed: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open".
Here is a more detailed log of the error:
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e327a87>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5caece52>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1fc5b16f>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1901cbeb>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e354641>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5624c1f3>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Aug 22 13:01:24 elastic7-logstash logstash[980]: [2019-08-22T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x78945d6f>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
On Kibana:
When I run the command:GET _cat/health?v
It gives me the following output:
epoch timestamp cluster status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1566505132 20:18:52 dev-cluster yellow 1 1 520 520 0 0 511 9 1.1s 50.4%
Currently, I have 504 indices open.
If I close the indexes, it helps me to recover from this issue temporally. Until more indexes are created and reach the limit.
Can you help me fix this issue? Thank you!