Timestamp and @timestamp problem with a big gap


(Michael Zoet) #1

Hello all,

I have a Logstash, Elasticsearch, Kibana, Logstash-forwarder setup running. Now I found a problem with the timestamp and @timestamp mismatch on some logfile types. I have no issue with syslog, Apache and PostgreSQL logs. But with all JAVA driven log types like Tomcat, Zuul and Eureka (and some microservices using Zuul and Eureka),

For example there is the following entry in logstash.log:

=====================
{
"message" => "2015-08-24 14:20:21.282 INFO 3167 --- [ReplicaAwareInstanceRegistry - RenewalThresholdUpdater] c.n.eureka.PeerAwareInstanceReg
istry : Current renewal threshold is : 64",
"@version" => "1",
"@timestamp" => "2015-01-24T13:20:21.282Z",
"file" => "/var/log/eureka/eureka-server.log",
"host" => "microgw-01",
"offset" => "5730664",
"server_location" => "01",
"type" => "eureka",
"fqdn" => "microgw-01.example.com",
"role" => "microgw",
"stage" => "production",
"server_type" => "XL",
"timestamp" => "2015-08-24 14:20:21.282",
"eureka_log_level" => "INFO",
"eureka_process_id" => "3167",
"eureka_thread_name" => "ReplicaAwareInstanceRegistry - RenewalThresholdUpdater",
"eureka_java_class_name" => "c.n.eureka.PeerAwareInstanceRegistry : Current renewal threshold is ",
"eureka_log_message" => "64"
}

I am really wondering why I have a difference of 7 months and one hour in timestamp and @timestamp. Maybe someone can give me hints or documentation links to sort this?

My configuration details for this logfile type in logstash.conf:

===================
...
if [type] =~ "eureka" {
multiline {
patterns_dir => "/etc/logstash/patterns"
pattern => "^%{MICROSERVICETIME}"
negate => true
what => "next"
}
grok {
patterns_dir => "/etc/logstash/patterns"
match => [ "message", "%{EUREKALOG}"]
}
date {
match => [ "timestamp", "YYYY-MM-DD HH:mm:ss.SSS" ]
timezone => "Europe/Berlin"
}
}
...

And the extra patterns:

===================

MICROSERVICETIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})

EUREKALOG ^%{MICROSERVICETIME:timestamp}%{SPACE}%{LOGLEVEL:eureka_log_level}%{SPACE}%{NUMBER:eureka_process_id}%{SPACE}---%{SPACE}[%{GREEDYDATA:eureka_thread_name}]%{SPACE}%{GREEDYDATA:eureka_java_class_name}%{SPACE}:%{SPACE}%{GREEDYDATA:eureka_log_message}$

===================

I hope someone can help me on this because at the moment "I do not see the forest for the trees" :wink: on this.

Thanks,
Michael


(Magnus B├Ąck) #2

I can't explain why this happens, but the DD in your pattern actually means "day of year" and not "day of month", and for whatever reason this changes how the preceding MM is interpreted. The pattern YYYY-MM-dd HH:mm:ss.SSS works for me.


(Michael Zoet) #3

Oh yeah... Thanks a lot! That was the problem and now it works like a charm.


(system) #4