Parsing timestamp from file and put in @timestamp

Elastic Stack Logstash
1

Hi all.
I use folowing scheme to collect logs: filebeat -> logstash -> graylog
So i have problem with parsing date and put it to the @timestamp

example of log:

20161012T082829Z|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|a54ca74edj5cajdb208i7e6bgf08846|https://xxx.ru|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://xxx.ru/idp|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_4675f53ddc3239b44bbff0e10a5539d7|salnikova-oy-130408|urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport|smsAuth,phone,principalId,group|salnikova-oy-130408|_b088e4ce70c78c59979fa0e8fe98e41f

in logstash i use this config:

filter {
if [type] == "idp" {
grok {
match => { "message" => "%{MY2_TIMESTAMP_ISO8601:idp_timestamp}Z|%{IDP:method}|%{IDP:hash1}|%{IDP:domain}|%{IDP}|%{IDP}|%{IDP}|%{IDP:hash2}|%{IDP:login}|%{IDP}|%{IDP}|%{IDP}|%{GREEDYDATA:hash3}" }
overwrite => [ "short_message" ]
}
date {
match => ["idp_timestamp", MY2_TIMESTAMP_ISO8601]
target => "@timestamp"
}

patterns file:

IDP [^|]+
MY2_TIMESTAMP_ISO8601 %{YEAR}%{MONTHNUM}%{MONTHDAY}T%{HOUR}%{MINUTE}%{SECOND}

so timestamp from log parsed to idp_timestamp field , but dont get into @timestamp
Where is my mistake? Or what i do wrong.

2

The date filter doesn't use grok patterns. This probably works:

date {
  match => ["idp_timestamp", "yyyyMMdd'T'HHmmss'Z'"]
}

Why not use a csv filter to parse each line?

3

thx Magnus!
it works but in another case i use this

date {
  match => ["idp_timestamp", ISO8601]
  target => "@timestamp"

}

and it also works, but you said that data filter doesn't use grok patterns.

one more question i parsed log which use UTC time , how can i add +3 hours ?

4

The @timestamp field is always UTC.

The date filter's timezone option is useful if the dates being parsed don't include a timezone and their actual timezone is different from the timezone of the machine where Logstash runs.

5

do i need to add
timezone => "Europe/Moscow"
in date ?

6

Not if the system timezone of the machine where Logstash runs is Europe/Moscow.

7

thank you ! we have changed timezone log timestamp in jetty ))