Problem parsing a datestring - Grok Debugger shows no problem

Elastic Stack Logstash
1

Example Logfile Entry which works perfectly in the Grok Debugger:

2018-11-21 08:15:54,809 WARN [com.xxx.mediaserver.tools.maintenance.FileMaintenance] (Thread-6) Not removing directory C:\xxx\mediaserver\import_export\SUPERVISOR\Downloads\packaged_ko_40186264_40186283 because we're not sure if it was created by us (directory name mismatch).

Here's the identification string:

%{DATE:date}%{SPACE}%{TIME:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}\[(?<class>.+?(?=]))\]%{SPACE}\((?<thread>.+?(?=\)))\)%{SPACE}(?<logmessage>.*)

And here's the logstash part:

grok { match => { "message" => "%{DATE:date}%{SPACE}%{TIME:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}[(?.+?(?=]))]%{SPACE}((?.+?(?=))))%{SPACE}(?.*)" } }
mutate { add_field => { "access_time" => "%{date} %{timestamp}" } }
date { match => [ "access_time", "YY-MM-dd HH:mm:ss,SSS" ] }
mutate {
remove_field => [ "SPACE" ]
remove_tag => [ "beats_input_codec_plain_applied", "monitor", "apache" ]
add_tag => [ "mediaserver" ]

In the field access_time i have this:

%{month} %{day} %{year} %{hour}:%{minute}:%{seconds}

so it seems, date parsing fails.

I also have this in field thread which makes no sense:

Thread-6) Not removing directory C:\xxx\mediaserver\import_export\SUPERVISOR\Downloads\packaged_ko_40186264_40186283 because we're not sure if it was created by us (directory name mismatch

because my regexp is parsing for the finishing ")" which works perfectly in Grok Debugger and ALSO in another part of the logstash config, which i post below:

This works perfectly:

  else if [quelle] == "wildfly-main" {
    grok { match => { "message" => "%{DATE:date}%{SPACE}%{TIME:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}\[(?<class>.+?(?=:)):%{INT:linenumber}\]%{SPACE}\((?<thread>.+?(?=\)))\)%{SPACE}(?<logmessage>.*)" } }
	mutate { add_field => { "access_time" => "%{date} %{timestamp}" } }
	date { match => [ "access_time", "YY-MM-dd HH:mm:ss,SSS" ] }
	mutate { 
	  remove_field => [ "SPACE", "date", "timestamp", "message" ] 
	  remove_tag => [ "beats_input_codec_plain_applied", "monitor", "apache" ]
	  add_tag => [ "wildfly" ]
    }
  }
2

Any reason that this part is different to this:

3

copy&paste error. this is the correct part in the logstash-config:

  else if [quelle] == "mediaserver-main" {
    grok { match => { "message" => "%{DATE:date}%{SPACE}%{TIME:timestamp}%{SPACE}%{WORD:loglevel}%{SPACE}\[(?<class>.+?(?=]))\]%{SPACE}\((?<thread>.+?(?=\)))\)%{SPACE}(?<logmessage>.*)" } }
	mutate { add_field => { "access_time" => "%{date} %{timestamp}" } }
	date { match => [ "access_time", "YY-MM-dd HH:mm:ss,SSS" ] }
	mutate { 
	  remove_field => [ "SPACE" ] 
	  remove_tag => [ "beats_input_codec_plain_applied", "monitor", "apache" ]
	  add_tag => [ "mediaserver" ]
    }
  }

So yes, i double and tripple checked that the correct string (which works in Grok Debugger) is also in the Logstash-Config. I am unable to find out why it doesn't work. I have another else if block with almost the exact same string which works like explained :frowning:

4

Testing your Grok only it works for me:

image
image.png1343×497 63.2 KB

I will try the rest of the filter tomorrow when I have more time.

5

yes, that's the funny thing. The second part in the logstash almost parses the same and it works aswell. I don't get it, try to solve it since 2-3 days.

What i did additionally:

  • Completely deleted indices in Elastic
  • Rebuild fields in Elastic
  • Reconfigured positioning in Logstash

Still the same problem :confused:

6

fixed the problem myself.
And now comes the funny part:

I just changed the position of the "else if" block in the logstash config. Nothing else.
Since then it works like expected.

I even copied it back to the original position (last one) and it still works.

Sometimes strange things happen and resolve themselves ...

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.