Kibana 7.4.0 all indices vanish at the end of each day

Elastic Stack Kibana
1

I have been using ELK in Docker for several years now, with zero issues. I am currently running a couple of entirely separate Dockerized ELK stacks, on entirely different servers running different environments, and am experiencing the same issue on both.

I'm using Kibana to access logs from various pieces of software, with each day having its own index. Generally it's useful to keep these indices for a month, then remove them. Unfortunately each morning when I log on the index from the previous day is gone, and only the index created on the current day is present. This applies to every single different index I have. Everything shows as healthy, there's plenty of hard drive space and free memory, but all past indices are gone without a trace.

I have looked into this extensively without any success. There was no index lifecycle set up at any point on either of the ELK stacks, so it would appear to be either something that happens by default in Kibana 7.4.0 or something that is being caused by the docker-compose file.

The docker-compose.yml file I'm using is as below.

    version: '3.7'

    services:

      # Elasticsearch Docker Images: https://www.docker.elastic.co/
      elasticsearch:
        image: docker.elastic.co/elasticsearch/elasticsearch:7.4.0
        container_name: elasticsearch
        environment:
          - xpack.security.enabled=false
          - discovery.type=single-node
          - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
        ulimits:
          memlock:
            soft: -1
            hard: -1
          nofile:
            soft: 65536
            hard: 65536
        cap_add:
          - IPC_LOCK
        volumes:
          - esdata1:/usr/share/elasticsearch/data
        ports:
          - 9200:9200
          - 9300:9300
        restart: always

      kibana:
        container_name: kibana
        image: docker.elastic.co/kibana/kibana:7.4.0
        environment:
          - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
        ports:
          - 5601:5601
        depends_on:
          - elasticsearch
        restart: always

    volumes:
      esdata1:
        driver: local
2

Hi @Josh_Creek welcome to the community.

Indeed strange... The mystery is a foot.

These appear to be single node clusters?

Perhaps run the cluster stats and see how long the nodes have been up. You can see the JVM uptime. Curious to see if they are getting restarted each day.

I did just run into something somewhat similar (but different) elsewhere where a new security tool / scan was killing processes it did not recognize.

3

Hi @stephenb , thank you for your response!

They are indeed single node clusters. Unlike the full product search engine system I used to run with a multi-node cluster, these are just serving as debugging and error log tools, so a single node is sufficient for my needs.

Having run the cluster stats it matches my expectations - I last rebooted the server almost a week ago, and the milliseconds here equate to 6 days and 9 hours, give or take some minutes, so it doesn't appear that the nodes are getting restarted each day.

"jvm" : {
      "max_uptime_in_millis" : 552759923,
}

Do you have any other suggestions? There's no other tools running on the machine, it's literally just a headless Linux box running ELK in Docker for convenience.

4

Have you looked at your data folder /usr/share/elasticsearch/data to see if anything is happening there?

Like possibly it being wiped out every day. Might be able to tell by looking at the folder/file creation times. I am not sure at all and just throwing out a guess. :slight_smile:

5

Interesting.... that would have been too easy.

What tools are you using to ingest the data?

What does the index naming / creation strategy look like? What is the config?

A thought today manually reindex some of that data into another index with a completely different name scheme ... No date etc. Let's see if it is there tomorrow. And or just create some other indices that don't follow your naming pattern let's see if they survive the night.

Also take a look at the data under the esdata directory check to see if any of the files are older than 24 hours ... There should be some ...

I suspect these are also unsecured clusters so anyone / process could run a DELETE /* not likely.. but not impossible.

6

Hi @aaron-nimocks - I've looked in the data folder in the elasticsearch container and the files haven't been modified since the server was last rebooted, but that was a good shout!

7

What tools are you using to ingest the data?

It varies across different coding languages, but mostly in C# projects it's NLog and in Node.js projects it's Winston.

What does the index naming / creation strategy look like? What is the config?

Generally it's in the format logs-applicationname-YYYY-MM-DD, which is specified in the code for each project. Each day it logs into the correctly dated index, creating it if it doesn't exist.

This has been the case since using an early version of Elasticsearch 6.something and has always worked prior to setting up this version.

A thought today manually reindex some of that data into another index with a completely different name scheme ... No date etc. Let's see if it is there tomorrow. And or just create some other indices that don't follow your naming pattern let's see if they survive the night.

I'll give that a go and see what happens.

Also take a look at the data under the esdata directory check to see if any of the files are older than 24 hours ... There should be some ...

Aye, there's a few that are several days old.

I suspect these are also unsecured clusters so anyone / process could run a DELETE /* not likely.. but not impossible.

They are, however it's on a private network inaccessible by any other users, so hopefully this isn't the issue.

8

Is there a particular reason 7.4.0
... technically the final stable is 7.4.2 if you settle on 7.4 that is what you should use.

Also perhaps try 7.10.2 just to see Although I doubt it is a ES version thing

So is each cluster for a single app or is there multiple indices / apps on each cluster?

Are you using an index template?

If you go back to the 6 version does this still happen? The big change ES 6-7 was the removal of mapping types.

Any code changes to the ingestors?

Yeah a doubt something is intentionally deleting but a malformed POST could

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.