@timestamp not matching date parsed, apache logs

Elastic Stack Logstash
1

Hello all, I'm new to log stash and probably this is a simple mistake on my part. I have a pile of old apache combined logs I want to put into elasticsearch, using logstash. I found that logstash was was not properly translating the date field into @timestamp.

I think I have a very basic filter setup:

grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
geoip { source => "clientip" }
useragent { source => "agent" }

However the output of @timestamp doesn't make any sense to me. For an input like [30/Aug/2015:21:44:25 -0500] I get the following out:

"@timestamp" => "2015-08-31T02:44:25.000Z",
"timestamp" => "30/Aug/2015:21:44:25 -0500",

Looks like a timezone offset problem somewhere, but I haven't figure out where. The original log is printed in central US time, my log stash is in US eastern time. I tried using the date{} "locale" and "timezone" fields, setting TZ shell variable. I verified my server (CentOS) is properly set to US/New_York.

Any thoughts? I'd like @timestamp to match exactly the apache log date ... or worst case have it be 1hr offset between eastern and central time.

thanks,

2

The ´@timestamp` field stores the time in UTC (you can tell from the "Z" at the end). Do not attempt to change this. Your presentation layer should make timezone adjustments as necessary. Kibana does.

If your log messages don't contain a timezone offset and the logs are produced in another timezone than where Logstash runs you'll want to set the date filter's timezone parameter to the log origin timezone.

3

What initially confused me was that I was trying to import 6007 log lines from Sep 30, and found logstash splitting this data across 2 indexes of 9/30 and 9/31. Hence my question, and I wasn't sure what tools like kibana would be using for date representation

Thanks for the quick reply! I'll ingest my data and leave things as is.