I have been using ELK in Docker for several years now, with zero issues. I am currently running a couple of entirely separate Dockerized ELK stacks, on entirely different servers running different environments, and am experiencing the same issue on both.
I'm using Kibana to access logs from various pieces of software, with each day having its own index. Generally it's useful to keep these indices for a month, then remove them. Unfortunately each morning when I log on the index from the previous day is gone, and only the index created on the current day is present. This applies to every single different index I have. Everything shows as healthy, there's plenty of hard drive space and free memory, but all past indices are gone without a trace.
I have looked into this extensively without any success. There was no index lifecycle set up at any point on either of the ELK stacks, so it would appear to be either something that happens by default in Kibana 7.4.0 or something that is being caused by the docker-compose file.
The docker-compose.yml file I'm using is as below.
version: '3.7'
services:
# Elasticsearch Docker Images: https://www.docker.elastic.co/
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
container_name: elasticsearch
environment:
- xpack.security.enabled=false
- discovery.type=single-node
- "ES_JAVA_OPTS=-Xms1g -Xmx1g"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
cap_add:
- IPC_LOCK
volumes:
- esdata1:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
restart: always
kibana:
container_name: kibana
image: docker.elastic.co/kibana/kibana:7.4.0
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- 5601:5601
depends_on:
- elasticsearch
restart: always
volumes:
esdata1:
driver: local
