Timestamp off(delay) 5 hours

Elastic Stack Logstash
1

Running version 6.2.4. Multiple logstash and each for different log type, like messages, secure... Index for messages log is off 5 hours. Everything else is correct. All logstash.conf files are similar. Here is the filter part of logstash.conf for messages log.

filter {
  if [type] == "messages" {
	grok {
	  match => { "message" => [ "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}",
							  "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{GREEDYDATA:syslog_message}" ] }
	  add_field =>  {
		"received_at" => "%{@timestamp}"
		"received_from" => "%{host}"
	  }
	}
	date {
	  match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
	}
	mutate {
	  replace => { "syslog_timestamp" => "@timestamp" }
	}
  }
}

Thanks,
netvmdb

2

In kibana,

@timestamp: November 9th 2018, 05:49:03.000
message: Nov 9 10:49:03

All other indices have correct timestamp, which the @timestamp is same as the time from message. But this one is different.

3

Timestamps in Elasticsearch are always in UTC timezone. Kibana can adjust to the local timezone in the UI, but does not alter the source document. Are you by any chance located in a place that is 5 hours off UTC?

4

I am in east time zone. Didn't set timezone either in Elasticsearch nor in Kibana.
When I create index pattern in Kibana, after enter index pattern, I select "@timstamp" from dropdown under "Time Filter field name".

In logstash.conf, I extract timestamp from message and assign it to "syslog_timestamp". Then replace "@timestamp" with value from "syslog_timestamp". I have several other type of logs, they all match @timestamp with timestamp from message. Only this index it doesn't match.

5

If the timestamp is not in the local timezone when logs are generated, you may need to specify timezone in the date filter. I think Logstash by default assumes the timestamp is in the same timezone as the host where it is being processed.

6

Could you show me how to specify timezone in date filter? My logstash.conf is on top of post.
Thanks.

7

I am not at a computer now, but I believe there should be an example in the documentation.

8

I got it.
timezone => "America/New_York"

Thanks for your help, Christian.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.