How to correctly parse and index dates into timestamp that have no offset? (Post title renamed)

Elastic Stack Logstash
1

Hi,

Hopefully this is a simple one.

I have a rather basic Logstash config to read nginx apache and error logs and push them to an index. It's all working OK except the date is incorrect for my error log file.

access log settings for timestamp which is working fine:
<<
date {
match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en

error log settings for timestamp which gives incorrect timestamp.
<<
date {
match => [ "err_time" , 'YYYY/MM/dd HH:mm:ss' ]
locale => en
}

Input from logfile:
2016/05/03 23:02:47

Creates the following in table view
@timestamp = May 4th 2016, 00:02:47.000
err_time = May 4th 2016, 00:02:47.000

And the following in the JSON document for the log entry.
<<
"@timestamp": "2016-05-03T23:02:47.000Z",
"err_time": "2016/05/03 23:02:47",

"fields": {
"err_time": [
1462316567000
],
"@timestamp": [
1462316567000
]
},

As you can see, the hour is being stripped off/ignored and the minutes are being used as the hour.

I'd really appreciate a steer on this, apologies in advance if I've missed something obvious in the documentation.

Cheers,

Mark.

2

The date filter creates UTC timestamps, which is the standard in Elasticsearch. Depending on which time zone you are in, an offset is therefore expected.

3

Thanks for the pointer. Before reading your reply I tried extending my mapping to support the date format.
<<
properties": {
"@timestamp": {
"type": "date",
"format": "YYYY/MM/dd HH:mm:ss||strict_date_optional_time||epoch_millis"

I also added the following the timezone thinking it might help

<<
date {
match => [ "err_time" , 'YYYY/MM/dd HH:mm:ss' ]
locale => en
timezone => "UTC"
}

Should I use add_field for the offset being one is not included in the source logfile? Are you able to provide example syntax for inserting the offset for dates that do not contain them?

Many thanks in advance.

Mark.

4

Post renamed to be more relevant to the problem as this was nothing to do with handling of multiple date formats and related to missing offset.

Thanks,

Mark.

5

Anyone able to help on this?

Thanks,

Mark.

6

Moved to Logstash from Kibana as advised by folk in IRC. Any pointers would be appreciated.

Thanks.

7

Solved.

The fix was a simple as adding 'timezone => "Europe/London"' into my date config. Thanks for the help all, especially the guys in #logstash IRC channel (bjorn_ and kartwheel'.

Cheers,

Mark.