My flux capacitor is broke - I.E. Elasticsearch index is in the future

Elastic Stack Elasticsearch
1

My cluster has been suffering for a while. It will go through the rebuild process and then it will end up in a red state. In diagnosing the issue, I looked through the indexes. I was amazed to find a bunch of indexes with future dates:

"logstash-syslog-events-2015.11.29": {
    "settings": {
        "index": {
            "creation_date": "1425324264913",
            "routing": {
                "allocation": {
                    "disable_allocation": "false"
                }
            },
            "uuid": "1Ftatk9GS5eNM07sXHwrHw",
            "number_of_replicas": "1",
            "number_of_shards": "5",
            "refresh_interval": "5s",
            "version": {
                "created": "1040499"
            }
        }
    }
},

Any recommendations on what could be doing this, or is this a huge problem? The dates on the server are correct and are responding properly with NTP.

2

The evidence certainly suggests that Logstash has received messages whose parsed @timestamp was 2015-11-29. What messages are in that index? Do you still have the original log files so you can check the source data? Or, if you keep the raw input message around in Elasticsearch that obviously works too.

3

So that being said, Elasticsearch is just doing what it is supposed to do and creating an index based on what it is receiving. I can accept that.

4

It's Logstash that creates the indexes based on the @timestamp field, but yes. Garbage in, garbage out.