I'm using ELK for Centralized logging and i'm facing DataLoss while processing 5lakh logs(kibana hits),
- (Logstash (Shipper&Indexer same instance)) Machine 1 -> (ElasticSearch -> Kibana) Machine 2
Scenario for DataLoss
- Logstash started reading log files with 5lakh logs and i'm able to see kibana hits increasing.
- While reading, ElasticSearch goes unavailable due to network issue between Machine 1 & 2.
- I have configured Logstash output, to retry for 10mins (retry count 120 times & interval 5secs).
- Why am I facing data loss in this scenario?
- In SinceDB file, What will be the offset position ? (position of logs read successfully / position of logs reached elastic search successfully)
- How to handle this scenario(ElasticSearch not available) without data loss ?