Timestamp issue in elasticsearch!

Elastic Stack Elasticsearch
1

No idea, why elasticsearch have a 4 hours delay in storing logs. In what basis it retrives the date settings ?.

When I restarted the elasticsearch cluster and checked the logs, timestamp printed on the logs too have delayed timestamp value.

> [2016-08-03 02:17:29,117][INFO ][http                     ] [node2] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/172.20.1.100:9200]}
> [2016-08-03 02:17:29,118][INFO ][node                     ] [node2] started
> ^C
> [root@Srv1 ~]# date
> Wed Aug  3 06:18:24 GMT 2016
> [root@Srv1 ~]#

In logs timestamp printed as "2016-08-03 02:17:29". But the server has a timestamp value of "Wed Aug 3 06:18:24 GMT 2016"

Please someone share your inputs. Thanks in Advance.

2

Have you convert the timestamp to UTC time ?? You can convert the timestamp to UTC first and then index the data to es .

3

fyi, the problem only with this node. I have one more node which doesn't have any issue.

I have a 2 node cluster setup with central logstash server to push data to nodes.

Actual Setup :
Logstash-server ------> Node 1
|
Node 2

Last night , Node 1 server has some network issue. So I just point the logstash server to "Node 2". After a while, I noticed all my logs are stored with 4 hours delay. means, my current logs will appear in kibana dashboard after 4 hours.

My suspection is with elasticsearch, As you can see the same delayed timestamp value in elasticsearch log file.

4

Any suggestions ?. Still I'm unable to fix this issue.

5

when you build timestamp do you use +-HH:MM at the end of your timestamp to set time in UTC according to your location ?
for example : "2016-08-03 02:17:29+02:00"

6

Sorry , I didn't get your point.

I would like to know , why the elasticsearch timestamp is set to different one [ with 4 hours delay] . even my server is set with correct value. From where it gets this value ?.

How can I fix this timestamp issue on this node without disturbing the cluster ?.

7

still I'm unable to fix the issue. Anyone can help on this?