Could not index event to elasticsearch maximum shards open

Elastic Stack Elasticsearch
1

Hi
I have problem with Elasticsearch and Logstash. Here is the detail of log:

Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e327a87>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5caece52>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1fc5b16f>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,828][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x1901cbeb>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x7e354641>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x5624c1f3>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}
Apr 16 13:01:24 elastic7-logstash logstash[980]: [2021-04-16T13:01:24,829][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"winlogbeat-7.3.0-2019.08.22", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x78945d6f>], :response=>{"index"=>{"_index"=>"winlogbeat-7.3.0-2019.08.22", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"validation_exception", "reason"=>"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [999]/[1000] maximum shards open;"}}}}

And I see at Stack management > Index management it has many index from the time before I installed the ELK. Image below
image
I dont know why and how to fix this. Plz help me

2

Welcome!

You can't have more than 1000 shards per node.

Here I think you have only one node.

So either you start 2 more nodes, or you remove some of the existing indices.

3

Thank you. How to Logstash doesnt collect log from the past.

4

What is the output of:

GET /
GET /_cat/nodes?v
GET /_cat/health?v
GET /_cat/indices?v

If some outputs are too big, please share them on gist.github.com and link them here.