Hi, we are using ELK stack for monitoring our applications. However, it's been quite weird to us that logs kept disappearing after a few hours in kibana.
The pattern was like:
Deployed applications -> logs appeared in kibana -> a few hours -> logs disappeared in kibana-> deployed applications -> logs appeared in kibana -> logs disappeared after a few hours ......
Everytime the logs disappeared in Kibana, we could see errors in kibana.log:
{"type":"log","@timestamp":"2021-04-23T11:53:47Z","tags":["error","elasticsearch","admin"],"pid":9236,"message":"Request error, retrying\nPOST http://localhost:9200/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms => socket hang up"}
We only have one elasticsearch node in cluster and each index has default 5 primary shards.
Do you have idea what could go wrong perhaps? or could you guide us where to debug this issue?
