When I am trying a load test suddenly I got following error on logstash logs and no records process afterward.
Before error occurred.
22569558 records were processed and 64.2gb space used by transactions-5k500s-2018.12.09 index.
I can see that index still exists in the elasticsearch db.
The test I am running is 5000 tps for 2.4 hours in a single node elasticsearch setup, it has 8 out of 16GB memory allocated.
Dec 09 07:06:45 nuwans-igw-2018q1-elk-ls-1 logstash[2489]: [2018-12-09T07:06:45,039][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["index", {:_id=>nil, :_index=>"transactions-5k500s-2018.12.09", :_type=>"doc", :_routing=>nil}, #<LogStash::Event:0x7fc72d87>], :response=>{"index"=>{"_index"=>"transactions-5k500s-2018.12.09", "_type"=>"doc", "_id"=>"AmXJkWcBFKG0mZxQQH8v", "status"=>404, "error"=>{"type"=>"index_not_found_exception", "reason"=>"no such index", "index_uuid"=>"_na_", "index"=>"transactions-5k500s-2018.12.09"}}}}
Logstash 6.3.0