Logs disappeared after running applications for a few hours - ELK stack

Hi, we are using ELK stack for monitoring our applications. However, it's been quite weird to us that logs kept disappearing after a few hours in kibana.

The pattern was like:
Deployed applications -> logs appeared in kibana -> a few hours -> logs disappeared in kibana-> deployed applications -> logs appeared in kibana -> logs disappeared after a few hours ......

Everytime the logs disappeared in Kibana, we could see errors in kibana.log:

{"type":"log","@timestamp":"2021-04-23T11:53:47Z","tags":["error","elasticsearch","admin"],"pid":9236,"message":"Request error, retrying\nPOST http://localhost:9200/_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms => socket hang up"}

We only have one elasticsearch node in cluster and each index has default 5 primary shards.

Do you have idea what could go wrong perhaps? or could you guide us where to debug this issue?

Welcome to our community! :smiley:

What version are you running?

Can you elaborate a little more on this part? How do you know it's disappearing?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.