Logstash creating index on different dates

Elastic Stack Elasticsearch
1

Hello,

I am seeing indexes being created in elasticsearch on different dates. I see the dates being created ranging from 1969 upto 2017. I have manually cleared all the wrong dated indices and i happened to see the indexes being created again.

Elasticsearch is currently running on three node cluster and elasticsearch ports are not being available to outside users except localhost logstash. the date is set in GMT and being synced from rhel pool ntp servers.

[root@den-prod-logstash02 ~]# rpm -qa | egrep -i lastic
elasticsearch-1.7.3-1.noarch
rpm [root@den-prod-logstash02 ~]# rpm -qa | egrep -i logstash
logstash-1.5.2-1.noarch
[root@den-prod-logstash02 ~]#

ex:
health status index pri rep docs.count docs.deleted store.size pri.store.size
green open logstash-1974.04.16 3 1 2 0 23.2kb 11.6kb
green open logstash-2014.06.14 3 1 540 0 414.3kb 207.2kb
green open logstash-2016.01.16 3 1 18483634 0 12.6gb 6.3gb
green open logstash-2016.01.04 3 1 73568307 0 50.1gb 25gb
green open logstash-2014.07.12 3 1 270 0 294.8kb 147.4kb
green open logstash-2016.01.24 3 1 20536996 0 13.8gb 6.9gb
green open .kibana 1 1 223 2 267.3kb 133.6kb
green open logstash-2014.11.11 3 1 1 0 20.4kb 10.2kb
green open logstash-2016.01.18 3 1 24043037 0 17.4gb 8.7gb
green open logstash-2017.01.26 3 1 60302 0 43.4mb 21.7mb
green open logstash-2014.05.23 3 1 1 0 15.4kb 7.7kb
green open logstash-1970.01.04 3 1 1 0 17.4kb 8.7kb
green open logstash-2014.06.24 3 1 270 0 287.5kb 143.8kb
green open logstash-2016.01.12 3 1 39176717 0 27.8gb 13.9gb
green open logstash-2016.01.30 3 1 24341713 0 69.8gb 34.8gb
green open logstash-2016.03.07 3 1 70 0 130.6kb 68kb
green open logstash-2016.01.09 3 1 37277851 0 25gb 12.4gb
green open logstash-1970.03.22 3 1 3 0 40.8kb 20.4kb
green open logstash-2014.05.14 3 1 1 0 15.4kb 7.7kb
green open logstash-2016.07.25 3 1 53953 0 38.9mb 19.4mb
green open logstash-2014.05.19 3 1 1 0 15.4kb 7.7kb
green open logstash-2016.01.05 3 1 59851034 0 40.6gb 20.3gb
green open logstash-2017.01.31 3 1 1385 0 1.1mb 580.4kb
green open logstash-2014.11.15 3 1 273 0 462.8kb 231.4kb
green open logstash-2014.11.10 3 1 1 0 18kb 9kb
green open logstash-2016.07.12 3 1 1 0 12.7kb 6.3kb
green open logstash-2014.06.15 3 1 271 0 291.8kb 145.9kb
green open logstash-2014.05.13 3 1 270 0 381.6kb 190.8kb
green open logstash-2016.01.23 3 1 21686720 0 15gb 7.5gb
green open logstash-2016.01.26 3 1 36073787 0 26.6gb 13.3gb
green open logstash-2014.04.30 3 1 1 0 15.4kb 7.7kb
green open logstash-1970.01.07 3 1 2 0 12.3kb 6.1kb

Could someone please help why different timestamped indices are being created.

Thanks in advance,

2

By default Logstash creates indexes based on the timestamp in the @timestamp field, suggesting that the problem is with the input data or the parsing of the date in the input data.

1 Like
3

Hi @magnusbaeck,

Thank you for the reply, So, does it means the indexes are getting created as per the timestamp in input logs data from LS agents?

4

Yes, that's the default configuration of Logstash's elasticsearch output.

5

Thank you @magnusbaeck