Timestamp filter logstash

Elastic Stack Logstash
1

I am trying to display the time from the message log, but the load time from logstash is coming out.
The message looks like this:

9.17.20.121 - - [11/Oct/2021:00:00:24 +0300] 0.474 0.072 "POST /api/?AppType=1&AppVersion=4.8.7.3&AgentID=eRgdy-erfs&SectionName=GetPreviousMessagesByService HTTP/1.0" 200 20305 "-" "Mozilla/3.0 (compatible; Indy Library)" "-"
My filter.conf

input {
file {
        path => "/var/log/logstash/test.log"
        start_position => "beginning"
       }
}
filter {
grok {
match => { "message" => "%{IPORHOST:clientip}%{SPACE}(?:-|(%{WORD}.%{WORD}))%{SPACE}%{USER:id}%{SPACE}\[%{HTTPDATE:timestamp}\]%{SPACE}%{BASE16FLOAT:request_time}%{SPACE}%{BASE16FLOAT:request_time_upstream}%{SPACE}\"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:rawrequest})\"%{SPACE}%{NUMBER:response}%{SPACE}(?:%{NUMBER:bytes}|-)%{SPACE}%{QS:referrer}%{SPACE}%{QS:agent}%{SPACE}%{QS:forwarder}" }
remove_field => "message"
remove_field => "host"
remove_field => "path"
remove_field => "@version"
}
grok {
match => { "message" => "%{HTTPDATE:logtimestamp}" }
}
date {
match => [ "logtimestamp", "dd/MM/YYYY:HH:mm:ss Z" ]
target => "logtimestamp"
remove_field => [ "logtimestamp" ]
locale => "en"
timezone => "UTC"
     }
mutate {
                replace => { "logtimestamp" => "%{@timestamp}" } }
}

pic

"logtimestamp" must have date 11 / Oct / 2021: 00: 02: 24 +0300 and type "Date".
Help Please.

2

Hi,

The date match is not correct.
According to the documentation :
MM => two-digit month. zero-padded if needed. Example: 01 for January and 12 for
MMM => abbreviated month text. Example: Jan for January. Note: The language used depends on your locale. See the locale setting for how to change the language. December

Here the format of the mounth is Oct so you need to use MMM instead of MM

match => [ "logtimestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]

Remove the followings part of your file, you are removing the field who contains the timestamp print in your log and after you re-creat it to put it the logstash one. This make no sense.

remove_field => [ "logtimestamp" ]

mutate {
                replace => { "logtimestamp" => "%{@timestamp}" } }
}

You have a _grokparsefailure to.
You can't do a match on the field message directly after delete it.
That gives you a _grokparsefailure when the first filter is successful

Cad.

1 Like
3

If your first grok works then you delete the [message] field. That makes the second grok and the date filter no-ops, since the fields they are trying to use do not exist. Even if the date filter did work, it would remove_field the value it had parsed, and then the mutate filter would replace it.

4

Your advice helped, thanks a lot!

5

Your hint helped me understand my mistake, thanks!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.