Hello all,
I have a Logstash, Elasticsearch, Kibana, Logstash-forwarder setup running. Now I found a problem with the timestamp and @timestamp mismatch on some logfile types. I have no issue with syslog, Apache and PostgreSQL logs. But with all JAVA driven log types like Tomcat, Zuul and Eureka (and some microservices using Zuul and Eureka),
For example there is the following entry in logstash.log:
=====================
{
"message" => "2015-08-24 14:20:21.282 INFO 3167 --- [ReplicaAwareInstanceRegistry - RenewalThresholdUpdater] c.n.eureka.PeerAwareInstanceReg
istry : Current renewal threshold is : 64",
"@version" => "1",
"@timestamp" => "2015-01-24T13:20:21.282Z",
"file" => "/var/log/eureka/eureka-server.log",
"host" => "microgw-01",
"offset" => "5730664",
"server_location" => "01",
"type" => "eureka",
"fqdn" => "microgw-01.example.com",
"role" => "microgw",
"stage" => "production",
"server_type" => "XL",
"timestamp" => "2015-08-24 14:20:21.282",
"eureka_log_level" => "INFO",
"eureka_process_id" => "3167",
"eureka_thread_name" => "ReplicaAwareInstanceRegistry - RenewalThresholdUpdater",
"eureka_java_class_name" => "c.n.eureka.PeerAwareInstanceRegistry : Current renewal threshold is ",
"eureka_log_message" => "64"
}
I am really wondering why I have a difference of 7 months and one hour in timestamp and @timestamp. Maybe someone can give me hints or documentation links to sort this?
My configuration details for this logfile type in logstash.conf:
===================
...
if [type] =~ "eureka" {
multiline {
patterns_dir => "/etc/logstash/patterns"
pattern => "^%{MICROSERVICETIME}"
negate => true
what => "next"
}
grok {
patterns_dir => "/etc/logstash/patterns"
match => [ "message", "%{EUREKALOG}"]
}
date {
match => [ "timestamp", "YYYY-MM-DD HH:mm:ss.SSS" ]
timezone => "Europe/Berlin"
}
}
...
And the extra patterns:
===================
MICROSERVICETIME %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})
EUREKALOG ^%{MICROSERVICETIME:timestamp}%{SPACE}%{LOGLEVEL:eureka_log_level}%{SPACE}%{NUMBER:eureka_process_id}%{SPACE}---%{SPACE}[%{GREEDYDATA:eureka_thread_name}]%{SPACE}%{GREEDYDATA:eureka_java_class_name}%{SPACE}:%{SPACE}%{GREEDYDATA:eureka_log_message}$
===================
I hope someone can help me on this because at the moment "I do not see the forest for the trees" 
on this.
Thanks,
Michael