Is it possible to recover logs that failed to transfer to ElasticSearch?

The following configuration is used to obtain the server's audit logs and forward them to Elasticsearch.

AuditBeat -> Logstash -> Elasticsearch

However, on Elasticsearch, the number of shards exceeded max_shards_per_node, and the following WARNING was output on Logstash and index creation failed.

Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"INDEX_NAME", :_type=>"_doc", :_routing=>nil}, #LogStash::Event:0x14f497d4], :response=>{"index"=>{"_index"=>"INDEX_NAME", "_type"=>"_doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Validation Failed: 1: this action would add [10] total shards, but this cluster currently has [6010]/[3000] maximum shards open;"}}}}

Currently, max_shards_per_node is set to a large value, so index creation is possible. However, the indices for logs of the periods that could not be transferred have not been created.

Is there any way to retrieve these lost logs?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.