The grok works very strangely in logstash

Elastic Stack Logstash
1

To all lovers of this hello!
In some excellent one noticed that my messages in logstash stopped parsing, an error appeared - _grokparsefailure. but no configuration changes were made.
Original messages

1.2.3.4 0 - Test [10/Dec/2021:09:27:35 +0200] "GET /sap(............................................................................................)/webif/notify/polling/SAR::dfyh778/notify.htm HTTP/1.1" 200 155
1.2.3.5 10 - - [10/Dec/2021:09:27:35 +0200] "POST /sap(............................................................................................)/webif/notify/polling/SAE::JHFJFJHJ3543J::202114.90/notify.htm HTTP/1.1" 200 155

part with the logstash config:

grok {
        patterns_dir => ["/etc/logstash/conf.d/patterns"]
        match => {"message" => "%{WEBDISPATCHERTPP}"}
}

Initially, the regular expression worked for these messages:

WEBDISPATCHERTPP %{IP:ip} %{NUMBER:duration} %{USER:remote_log_name} %{USER:ident} \[(?<timestamp>.*)\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

at some point these regular expressions stopped working in lostash, but at the same time everything is parsed perfectly in Grok Debugger in kibana

what is the problem? I tried to partially rewrite the parsing, but I cannot fully describe it, there is always an error in some area

2

Are those an example of the the messages that have _grokparsefailure?

3

Unfortunately yes.
but with this in the GROK debugger, with the help of this expression, everything is parsed perfectly

Now I painted it a little, with other regular expressions and more general ones, but I think this is not entirely correct. At the same time, the time is parsed from me only after - %{GREEDYDATA}.

maybe something is wrong with my version of logtesh or kibana? check with yourself.

4 
input { generator { count => 1 lines => [
    '1.2.3.4 0 - Test [10/Dec/2021:09:27:35 +0200] "GET /sap(............................................................................................)/webif/notify/polling/SAR::dfyh778/notify.htm HTTP/1.1" 200 155',
    '1.2.3.5 10 - - [10/Dec/2021:09:27:35 +0200] "POST /sap(............................................................................................)/webif/notify/polling/SAE::JHFJFJHJ3543J::202114.90/notify.htm HTTP/1.1" 200 155'
] } }
filter {
    grok {
        pattern_definitions => { "WEBDISPATCHERTPP" => '%{IP:ip} %{NUMBER:duration} %{USER:remote_log_name} %{USER:ident} \[(?<timestamp>.*)\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)' }
        match => {"message" => "%{WEBDISPATCHERTPP}"}
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

parses the lines just fine, so I do not think the problem is in the grok patterns.

5

Thank you for checking, and what is your version of Logstash?

6

In the configuration, we registered everything as you have in the example, my Parsig earned. But with PAINTS_DIR => ["/etc/logstash/conf.d/patterns"] does not work. While I will observe the work. very strange.