Parser for multiple line file

Elastic Stack Logstash
1

Hello experts!

I have the following scenario: I have a logfile with differente and multiple line log. There are at least 3 structures inside the file

I have done the grok parser, but when I execute in logstash, It does not work.

The filter file has the following structure:

filter {
if [type] == "pts" {
grok {
patterns_dir => ["C:/ELK/Logstash/logstash-2.3.4/pts_patterns"]
match => [
"message", "parser_for_line_1",
"message", "parser_for_line_2",
"message", "parser_for_line_3",
]
}
}
}

The log file has the following content:

PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCESS:PTS_ONLINE_D|PROCESS STATUS:POSTING
PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:POSTING_ENGINE|POSTING_MODE:TWO_MESSAGE_WITHOUT_RESERVE|OPERATION:2M_DEBITO|ERROR_CODE:-1|ERROR_DESC:ERROR
PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:PTS_RULE_ITERATOR|RULEID:BR-TX-ALL-023|ERROR_CODE:0|ERROR_DESC:

Please, let me know, if the approach is correct, because it doesn't work so far.

Thanks a lot

Kind regards.

Dario R.

2

Please show

  • the raw output from a stdout { codec => rubydebug } output for a message that isn't processed correctly (and presumably has a _grokparsefailure tag) and
  • your custom grok patterns.
3

Hello Magnus.

The raw output, doesn't show any error, but i have reviewed and I conclude the parser doesn't work fine because some variables are missing. This is the raw output:

Pipeline main started
{
          "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCESS:PTS_ONLINE_D|PROCESS STATUS:POSTING\r",
         "@version" => "1",
       "@timestamp" => "2016-09-15T13:24:12.288Z",
             "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt",
             "host" => "HRODRIGUEZ",
             "type" => "pts",
        "l_pts_id1" => "PTS_ID",
          "pts_id1" => "PI30286",
         "l_trxId1" => "TRXID",
           "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c",
        "l_process" => "PROCESS",
          "process" => "PTS_ONLINE_D",
    "l_processStat" => "PROCESS STATUS",
      "processStat" => "POSTING"
}
{
          "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:POSTING_ENGINE|POSTING_MODE:TWO_MESSAGE_WITHOUT_RESERVE|OPERATION:2M_DEBITO|ERROR_CODE:-1|ERROR_DESC:ERROR\r",
         "@version" => "1",
       "@timestamp" => "2016-09-15T13:24:13.062Z",
             "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt",
             "host" => "HRODRIGUEZ",
             "type" => "pts",
        "l_pts_id1" => "PTS_ID",
          "pts_id1" => "PI30286",
         "l_trxId1" => "TRXID",
           "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c",
        "l_process" => "PROCEDURE",
          "process" => "POSTING_ENGINE",
    "l_processStat" => "POSTING_MODE",
      "processStat" => "TWO_MESSAGE_WITHOUT_RESERVE"
}
{
          "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:PTS_RULE_ITERATOR|RULEID:BR-TX-ALL-023|ERROR_CODE:0|ERROR_DESC: \r",
         "@version" => "1",
       "@timestamp" => "2016-09-15T13:24:13.064Z",
             "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt",
             "host" => "HRODRIGUEZ",
             "type" => "pts",
        "l_pts_id1" => "PTS_ID",
          "pts_id1" => "PI30286",
         "l_trxId1" => "TRXID",
           "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c",
        "l_process" => "PROCEDURE",
          "process" => "PTS_RULE_ITERATOR",
    "l_processStat" => "RULEID",
      "processStat" => "BR"
}

the custom patterns are:

_PTS_LOG \[[A-Z]+\_[A-Z]+\]_
_POOL \([a-z]+\-[0-9]\-[a-z]+\-[0-9]\)_
_TRXID [a-z0-9-]+_
_FIELD_NAME [A-Z_\ ]+_
_RULE_ID [ (.\-\d\D)]+_
_ERROR_CODE [(\d\D)]{1,2}_
_ERROR_DESC [(\d\D)]+_
_PTS_ID [A-Z0-9_\ ]+_
_PROC [A-Z_\ ]+_
_POST [A-Z_\ ]+_
_OPERATION [A-Z0-9_\ ]+_

Finally, this is the filter part detailed:

filter {
	if [type] == "pts" {
		grok { 
			patterns_dir => ["C:/ELK/Logstash/logstash-2.3.4/pts_patterns"]
			match => [ 
			"message", "%{FIELD_NAME:l_pts_id1}:%{PTS_ID:pts_id1}\|%{FIELD_NAME:l_trxId1}:%{TRXID:trxid1}\|%{FIELD_NAME:l_process}:%{FIELD_NAME:process}\|%{FIELD_NAME:l_processStat}:%{FIELD_NAME:processStat}",
			"message", "%{FIELD_NAME:l_pts_id2}:%{PTS_ID:pts_id2}\|%{FIELD_NAME:l_trxId2}:%{TRXID:trxid2}\|%{FIELD_NAME:l_proc}:%{PROC:proc}\|%{FIELD_NAME:l_postmode}:%{POST:postmode}\|%{FIELD_NAME:l_operation}:%{OPERATION:operation}\|%{FIELD_NAME:l_errorcode}:%{ERROR_CODE:errorcode}\|%{FIELD_NAME:l_errordesc}:%{ERROR_DESC:error_desc}",
			"message", "%{FIELD_NAME:l_pts_id3}:%{PTS_ID:pts_id3}\|%{FIELD_NAME:l_trxId3}:%{TRXID:trxid3}\|%{FIELD_NAME:l_proc2}:%{PROC:proc2}\|%{FIELD_NAME:l_ruleid}:%{RULE_ID:ruleid}\|%{FIELD_NAME:l_errorcode2}:%{ERROR_CODE:errorcode}\|%{FIELD_NAME:l_errordesc}:%{ERROR_DESC:error_desc2}"
			]
		}
	}
}

Thanks in advance

regard

Dario R

4

So what's missing from the output?

5

The file has three kind of lines. Also, the grok filter has 3 messages each with a parser.
But the output only has the variables of one of message, that means the variables of the message parser are missing. I don't know why this happen, because in grok debugger all is fine.
Thanks

6

What probably happens is that the first grok expression matches all messages so the filter doesn't bother trying with the rest. Either list the expressions in reverse other with the most specific expression first or add a $ anchor at the end of each expression so that only exact matches are accepted.