Elasticsearch Indices showing wrong date

Elastic Stack Elasticsearch
1

Hi , MY elastic indices showing wrong date why like yellow open syslog-2018.08.08 2ERImxJkQIOGw0qImjMPlw 5 1 37 0 401.8kb 401.8kb

and secondly the status of my elastic shows yellow however its running good. its stand alone server no cluster.

curl -s -XGET http://localhost:9200/_cat/indices?v

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open syslog-2018.05.25 CoLPTab_QlaaS1nu81VSOA 5 1 2353549 0 560.3mb 560.3mb
yellow open syslog-2018.06.12 Nj74y0BuQPGAG4iyKv0mPQ 5 1 54845 0 23.8mb 23.8mb
yellow open syslog-2018.08.08 2ERImxJkQIOGw0qImjMPlw 5 1 37 0 401.8kb 401.8kb
yellow open syslog-2018.05.23 WsiAi47fQvqgnPb0D7AAqA 5 1 1777821 0 369mb 369mb
yellow open syslog-2018.05.18 4XF9CrGjRJG3hxoKuU95Bw 5 1 1573732 0 349.1mb 349.1mb
yellow open syslog-2018.05.31 jAQx--a4R5q9ivWhW5puTg 5 1 118580404 0 27.1gb 27.1gb
yellow open syslog-2018.06.08 YJT_KmX3QA-Z9Ha9ASyCMA 5 1 297562 0 94.2mb 94.2mb
yellow open syslog-2018.05.17 65T03amNTi-_UW6qZTevFQ 5 1 1452690 0 326.8mb 326.8mb
yellow open syslog-2018.06.01 DP466AlVTRCcr2l_ifQznQ 5 1 116901880 0 26.8gb 26.8gb
yellow open syslog-2018.06.02 -MJZuhD4Rc6EMw3jL4NZOw 5 1 115751616 0 26.3gb 26.3gb
yellow open syslog-2018.05.29 pGHCrl7jQNOocCCQdhznaA 5 1 122179799 0 28.1gb 28.1gb
yellow open syslog-2018.06.05 XR4RQ1kgR0qA1D5R0-oKrQ 5 1 112749112 0 25.5gb 25.5gb
yellow open syslog-2018.08.09 tBA4WWMIRwqjA3T_b6HB9w 5 1 7 0 65.9kb 65.9kb
yellow open syslog-2018.08.14 Sg4HJIBBQ-2J6bG3GhvH5Q 5 1 7 0 77.6kb 77.6kb
yellow open syslog-2018.05.26 2gEOG5gATGquTHvBD_SV_A 5 1 1922332 0 418.1mb 418.1mb
yellow open syslog-2018.08.07 ZOhU_2tpRt2VTrgiGP2AJg 5 1 124 0 316.6kb 316.6kb
yellow open syslog-2018.06.06 Yo5JxMPiTNql8fRgVpWj4g 5 1 117152425 0 26.6gb 26.6gb
yellow open syslog-2018.08.01 JnSoLyBcTBmFWOQ4z70HXQ 5 1 3 0 40.4kb 40.4kb
yellow open syslog-2018.05.21 3decyvlnSouNLZ-PGNojwA 5 1 1266415 0 284.4mb 284.4mb
yellow open syslog-2018.05.28 JzAbYe9fRW-HcWOtWwfkog 5 1 79822739 0 18gb 18gb
yellow open syslog-2018.08.05 mHNp5wigTqe8S9ZgxTBrBw 5 1 5 0 61.7kb 61.7kb
green open .kibana _m1x8oARSmm2fn-7bIN_JQ 1 0 2 0 11.1kb 11.1kb
yellow open syslog-2018.06.07 tIn4e7leQkyH39D1Z9BMOQ 5 1 43199023 0 10gb 10gb
yellow open syslog-2018.05.09 NrtGPt9ySTCdkVZNC4iy8g 5 1 1064483 0 262.5mb 262.5mb
yellow open syslog-2018.05.27 91Dn-5yGRp-TDNE2BIWKfw 5 1 1996440 0 441.1mb 441.1mb
yellow open syslog-2018.05.16 fEEK9l4JQzm5VuuzjEfNxw 5 1 1413379 0 315.6mb 315.6mb
yellow open syslog-2018.06.10 YS4GUKlQS5e2EUdyOoeRgw 5 1 294817 0 75.3mb 75.3mb
yellow open syslog-2018.08.06 YiJgS816Rjm4h8mz87xjUQ 5 1 17 0 205kb 205kb
yellow open syslog-2018.05.11 M7IPK0VpRQ2X6uyG7yksdw 5 1 1613951 0 337.1mb 337.1mb
yellow open syslog-2018.05.14 g43pFIvNTO6pYVPof3HJdA 5 1 1337746 0 304.3mb 304.3mb
yellow open syslog-2018.06.11 brLuEuJDTGG-_DZ1v5y0Cg 5 1 272586 0 77.3mb 77.3mb
yellow open syslog-2018.08.13 SQw3Ex-7Q22cYqVhuZbtow 5 1 17 0 192.9kb 192.9kb
yellow open syslog-2018.08.10 VQAMVknMRbmnvcxxCzzmYA 5 1 2 0 25.5kb 25.5kb
yellow open syslog-2018.05.20 GY4NNoiUTu6zR43r-CQwQg 5 1 1162194 0 264.8mb 264.8mb
yellow open syslog-2018.06.03 y6GgekbHSguCw1sw2EnT0w 5 1 115347226 0 26.1gb 26.1gb
yellow open syslog-2018.08.11 yR0A7FJWRmqrQPGsZ7h2GA 5 1 12 0 117.8kb 117.8kb
yellow open syslog-2018.05.10 hZCRQvlzR-iy2q0M5Z3zzA 5 1 1126974 0 266.8mb 266.8mb
yellow open syslog-2018.05.30 HlcJ3KECRUW5Ci4EcXteXA 5 1 119844954 0 27.4gb 27.4gb
yellow open syslog-2018.05.06 YeFJeMIYSbyz3kih70pM5Q 5 1 1495289 0 374.9mb 374.9mb
yellow open syslog-2018.05.08 uKACFzXISemD4nqjudgzFQ 5 1 1251212 0 321.7mb 321.7mb
yellow open syslog-2018.05.07 JCV3AELdQsqmfutdYxUNGQ 5 1 2032953 0 468.9mb 468.9mb
yellow open syslog-2018.05.24 avgw59nXRqWBYQzT6CulLA 5 1 2317887 0 565.4mb 565.4mb
yellow open syslog-2018.08.12 kAIEkjFwQBG-DUPfJHT26A 5 1 10 0 102kb 102kb
yellow open syslog-2018.05.12 NWP0VkmcSxaw-MdUT57XgQ 5 1 1047570 0 233.1mb 233.1mb
yellow open syslog-2018.05.13 noQ78WOpR-SmF2StUPYTwA 5 1 1116463 0 245.2mb 245.2mb
yellow open syslog-2018.05.22 NtEm05w3RLmfVsFZ6E9FjA 5 1 1311456 0 284.2mb 284.2mb
yellow open syslog-2018.06.09 eSCVqpKqQwu8KXOIW7Hteg 5 1 213965 0 60.5mb 60.5mb
yellow open syslog-2018.05.19 0bDz_FboRWavkFANDh7ISA 5 1 1172345 0 271.8mb 271.8mb
yellow open syslog-2018.06.04 aVo7oeDUTl2zXljdxGUQQw 5 1 116990998 0 26.6gb 26.6gb
yellow open syslog-2018.05.15 oNcfoaqdQ7ujcG5PsWA6Wg 5 1 1442143 0 326.4mb 326.4mb

2

If you only have one node in the cluster, Elasticsearch will not be able to allocate the replica, which results in a yellow state. You can fix this by setting the number of replicas to 0 or by adding nodes to the cluster.

It seems like quite few documents are generated with the incorrect date. have a look at the documents and see if you can tell where they are coming from. You may have a device with date incorrectly set.

3

I concur with @Christian_Dahlqvist about dates in the raw data likely being wrong. Fortinet is particularly bad in our environment. What makes it even more weird is that it isn't all logs from Fortinet, just a few. Fortinet also has date weirdness when sending Netflow.

1 Like
4

@Christian_Dahlqvist, thanks for the suggestion , i'll check fix it.

5

@Christian_Dahlqvist, can you point me the location where i can set this replica settings .. i was looking at elasticsearch.yml.

6

got it ...

index.number_of_replicas: 0 while doing this in elasticsearch.yml service is getting killed and showing elasticsearch dead but subsys locked hence u reverted the change and its running fine.

7

You have to update the index settings vi the REST API. You can do this in Kibana -> Dev Tools -> Console like this:

PUT thenameoftheindex/_settings
{
  "index" : {
    "number_of_replicas" : 0
  }
}

Since you are writing daily indices you will will want to update this same setting in the Index Template used to create them. If the Index Template is being managed by Logstash, you can edit the template and restart Logstash. If the template was loaded into Elasticsearch manually via the REST API, you can also use the API to update it. This can also be done in the console like this:

PUT _template/thenameofthetemplate
{
  "settings": {
    "number_of_replicas": 0
  }
}

I will apologize in advance if those are a little off. I am working from memory.

8

@rcowart, thnx for the advice i'll check on this.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.