Parser for multiple line file

rodher · September 14, 2016, 10:34pm

Hello experts!

I have the following scenario: I have a logfile with differente and multiple line log. There are at least 3 structures inside the file

I have done the grok parser, but when I execute in logstash, It does not work.

The filter file has the following structure:

filter {
if [type] == "pts" {
grok {
patterns_dir => ["C:/ELK/Logstash/logstash-2.3.4/pts_patterns"]
match => [
"message", "parser_for_line_1",
"message", "parser_for_line_2",
"message", "parser_for_line_3",
]
}
}
}

The log file has the following content:

Please, let me know, if the approach is correct, because it doesn't work so far.

Thanks a lot

Kind regards.

Dario R.

magnusbaeck · September 15, 2016, 5:39am

Please show

the raw output from a stdout { codec => rubydebug } output for a message that isn't processed correctly (and presumably has a _grokparsefailure tag) and
your custom grok patterns.

rodher · September 15, 2016, 1:39pm

Hello Magnus.

The raw output, doesn't show any error, but i have reviewed and I conclude the parser doesn't work fine because some variables are missing. This is the raw output:

Pipeline main started
{
          "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCESS:PTS_ONLINE_D|PROCESS STATUS:POSTING\r",
         "@version" => "1",
       "@timestamp" => "2016-09-15T13:24:12.288Z",
             "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt",
             "host" => "HRODRIGUEZ",
             "type" => "pts",
        "l_pts_id1" => "PTS_ID",
          "pts_id1" => "PI30286",
         "l_trxId1" => "TRXID",
           "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c",
        "l_process" => "PROCESS",
          "process" => "PTS_ONLINE_D",
    "l_processStat" => "PROCESS STATUS",
      "processStat" => "POSTING"
}
{
          "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:POSTING_ENGINE|POSTING_MODE:TWO_MESSAGE_WITHOUT_RESERVE|OPERATION:2M_DEBITO|ERROR_CODE:-1|ERROR_DESC:ERROR\r",
         "@version" => "1",
       "@timestamp" => "2016-09-15T13:24:13.062Z",
             "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt",
             "host" => "HRODRIGUEZ",
             "type" => "pts",
        "l_pts_id1" => "PTS_ID",
          "pts_id1" => "PI30286",
         "l_trxId1" => "TRXID",
           "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c",
        "l_process" => "PROCEDURE",
          "process" => "POSTING_ENGINE",
    "l_processStat" => "POSTING_MODE",
      "processStat" => "TWO_MESSAGE_WITHOUT_RESERVE"
}
{
          "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:PTS_RULE_ITERATOR|RULEID:BR-TX-ALL-023|ERROR_CODE:0|ERROR_DESC: \r",
         "@version" => "1",
       "@timestamp" => "2016-09-15T13:24:13.064Z",
             "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt",
             "host" => "HRODRIGUEZ",
             "type" => "pts",
        "l_pts_id1" => "PTS_ID",
          "pts_id1" => "PI30286",
         "l_trxId1" => "TRXID",
           "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c",
        "l_process" => "PROCEDURE",
          "process" => "PTS_RULE_ITERATOR",
    "l_processStat" => "RULEID",
      "processStat" => "BR"
}

the custom patterns are:

_PTS_LOG \[[A-Z]+\_[A-Z]+\]_
_POOL \([a-z]+\-[0-9]\-[a-z]+\-[0-9]\)_
_TRXID [a-z0-9-]+_
_FIELD_NAME [A-Z_\ ]+_
_RULE_ID [ (.\-\d\D)]+_
_ERROR_CODE [(\d\D)]{1,2}_
_ERROR_DESC [(\d\D)]+_
_PTS_ID [A-Z0-9_\ ]+_
_PROC [A-Z_\ ]+_
_POST [A-Z_\ ]+_
_OPERATION [A-Z0-9_\ ]+_

Finally, this is the filter part detailed:

filter {
	if [type] == "pts" {
		grok { 
			patterns_dir => ["C:/ELK/Logstash/logstash-2.3.4/pts_patterns"]
			match => [ 
			"message", "%{FIELD_NAME:l_pts_id1}:%{PTS_ID:pts_id1}\|%{FIELD_NAME:l_trxId1}:%{TRXID:trxid1}\|%{FIELD_NAME:l_process}:%{FIELD_NAME:process}\|%{FIELD_NAME:l_processStat}:%{FIELD_NAME:processStat}",
			"message", "%{FIELD_NAME:l_pts_id2}:%{PTS_ID:pts_id2}\|%{FIELD_NAME:l_trxId2}:%{TRXID:trxid2}\|%{FIELD_NAME:l_proc}:%{PROC:proc}\|%{FIELD_NAME:l_postmode}:%{POST:postmode}\|%{FIELD_NAME:l_operation}:%{OPERATION:operation}\|%{FIELD_NAME:l_errorcode}:%{ERROR_CODE:errorcode}\|%{FIELD_NAME:l_errordesc}:%{ERROR_DESC:error_desc}",
			"message", "%{FIELD_NAME:l_pts_id3}:%{PTS_ID:pts_id3}\|%{FIELD_NAME:l_trxId3}:%{TRXID:trxid3}\|%{FIELD_NAME:l_proc2}:%{PROC:proc2}\|%{FIELD_NAME:l_ruleid}:%{RULE_ID:ruleid}\|%{FIELD_NAME:l_errorcode2}:%{ERROR_CODE:errorcode}\|%{FIELD_NAME:l_errordesc}:%{ERROR_DESC:error_desc2}"
			]
		}
	}
}

Thanks in advance

regard

Dario R

magnusbaeck · September 15, 2016, 6:20pm

So what's missing from the output?

rodher · September 15, 2016, 7:07pm

The file has three kind of lines. Also, the grok filter has 3 messages each with a parser.
But the output only has the variables of one of message, that means the variables of the message parser are missing. I don't know why this happen, because in grok debugger all is fine.
Thanks

magnusbaeck · September 17, 2016, 5:35pm

What probably happens is that the first grok expression matches all messages so the filter doesn't bother trying with the rest. Either list the expressions in reverse other with the most specific expression first or add a $ anchor at the end of each expression so that only exact matches are accepted.