I am running the docker-elk stack for a small project. Without changing the default config I launched docker compose and the containers are all up and running and had been working fine for 3 months, but suddenly I find that all the data disappears after one day. Or rather each day I can only see todays logs. I'm collecting data from logstash and it is coming in fine. Also each day any saved searches in Kibana are gone and I have to make a new index.
I have spent a couple of hours looking at he logs in each container and don't see any errors but maybe I'm looking in the wrong place. The java applications themselves are not restarting, they all show the original launch time.
I suspect this is either a data retention setting I need to fix or something is rebooting each day.
What I'm looking for apart from generic trouble shooting is to check if the data is still there using elasticsearches web api and how to stop it from vanishing.