I'm running both Logstash server and elasticsearch in same machine.
After parsing the logs into ELS , I checked it through kibana where the data shown with 5 minutes delay. Then I found that the server time [ NTP] is not configured correctly that was running with 5 minutes delay. that's the reason why I saw the same delay in my kibana.
Strange here is ,after reset my server time with correct one, I couldn't see the data in correct time format. Still it shows with 5 seconds delay. Should I need to reset the time settings somewhere else
Server time:
[root@ip202 conf.d]# date
Fri Jul 22 12:11:08 GMT 2016
[root@ip202 conf.d]#
Logstash Output log:
"@timestamp" => "2016-07-22T12:06:49.161Z",
Where does @timestamp come from? Are you parsing it from a log file's contents or it Logstash populating it by itself with the current time? Please show the output of date | /opt/logstash/bin/logstash agent.
I got that @timestamp value from logstash.stdout log file and this timestamp value is being populated by logstash itself with current time.
[root@ip202 ~]# date | /opt/logstash/bin/logstash agent
Settings: Default pipeline workers: 2
Pipeline main started
{
"message" => "Sat Jul 23 10:52:15 GMT 2016",
"@version" => "1",
"@timestamp" => "2016-07-23T10:52:22.747Z",
"type" => "stdin",
"host" => "ip202.ip-51-255-232.eu"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
[root@ip202 ~]#
@magnusbaeck ,, Explaining my issue bit more clear
I'm using filebeat/logstash-forwarder to push the logs from my remote servers. for now, I've tried to push the logs from 2 remote servers which are running with different OS version.
1.) Centos 6.7 --> Here I'm using "Filebeat" .
2.) Centos 5.8 --> Here I'm using "logstash-forwarder". [ As filebeat doesn't support on this version]
I don't have any problem with parsing Centos5.8 server logs through logstash-forwarder. timestamp field is updated/logged properly in elasticsearch.
But, in CentOS6.7 , As mentioned earlier post , timestamp field has a 5 minutes delay .
Strange here is, If I stopped the filebeat and installed logstash-forwarder in CentOS6.7 machine, timestamp field updated properly and stored in Elasticsearch and can see the data in kibana without any delay.
I got confused here, Is timestamp field generated by filebeat ?. I though, this field value can be handled by logstash servers.
I tried again with filbeat. Still hit with same problem. Anyone have idea what is reason for this ?
1 Like