# Parser for multiple line file **URL:** \ **Category:** Logstash\ **Created:** [September 14, 2016, 10:34pm UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555 "2016-09-14T22:34:55Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rodher](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rodher](https://discuss.elastic.co/u/rodher)\ **Post date:** [September 14, 2016, 10:34pm UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/1 "2016-09-14T22:34:56Z")
Hello experts! I have the following scenario: I have a logfile with differente and multiple line log. There are at least 3 structures inside the file I have done the grok parser, but when I execute in logstash, It does not work. The filter file has the following structure: filter { if [type] == "pts" { grok { patterns\_dir =\> ["C:/ELK/Logstash/logstash-2.3.4/pts\_patterns"] match =\> [ "message", "parser\_for\_line\_1", "message", "parser\_for\_line\_2", "message", "parser\_for\_line\_3", ] } } } The log file has the following content: PTS\_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCESS:PTS\_ONLINE\_D|PROCESS STATUS:POSTING PTS\_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:POSTING\_ENGINE|POSTING\_MODE:TWO\_MESSAGE\_WITHOUT\_RESERVE|OPERATION:2M\_DEBITO|ERROR\_CODE:-1|ERROR\_DESC:ERROR PTS\_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:PTS\_RULE\_ITERATOR|RULEID:BR-TX-ALL-023|ERROR\_CODE:0|ERROR\_DESC: Please, let me know, if the approach is correct, because it doesn't work so far. Thanks a lot Kind regards. Dario R. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2016, 5:39am UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/2 "2016-09-15T05:39:32Z")
Please show - the raw output from a `stdout { codec => rubydebug }` output for a message that isn't processed correctly (and presumably has a `_grokparsefailure` tag) and - your custom grok patterns. ---
**Author:** ![rodher](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rodher](https://discuss.elastic.co/u/rodher)\ **Post date:** [September 15, 2016, 1:39pm UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/3 "2016-09-15T13:39:57Z")
Hello Magnus. The raw output, doesn't show any error, but i have reviewed and I conclude the parser doesn't work fine because some variables are missing. This is the raw output: ``` Pipeline main started { "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCESS:PTS_ONLINE_D|PROCESS STATUS:POSTING\r", "@version" => "1", "@timestamp" => "2016-09-15T13:24:12.288Z", "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt", "host" => "HRODRIGUEZ", "type" => "pts", "l_pts_id1" => "PTS_ID", "pts_id1" => "PI30286", "l_trxId1" => "TRXID", "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c", "l_process" => "PROCESS", "process" => "PTS_ONLINE_D", "l_processStat" => "PROCESS STATUS", "processStat" => "POSTING" } { "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:POSTING_ENGINE|POSTING_MODE:TWO_MESSAGE_WITHOUT_RESERVE|OPERATION:2M_DEBITO|ERROR_CODE:-1|ERROR_DESC:ERROR\r", "@version" => "1", "@timestamp" => "2016-09-15T13:24:13.062Z", "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt", "host" => "HRODRIGUEZ", "type" => "pts", "l_pts_id1" => "PTS_ID", "pts_id1" => "PI30286", "l_trxId1" => "TRXID", "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c", "l_process" => "PROCEDURE", "process" => "POSTING_ENGINE", "l_processStat" => "POSTING_MODE", "processStat" => "TWO_MESSAGE_WITHOUT_RESERVE" } { "message" => "PTS_ID:PI30286|TRXID:9a80fd8e-8946-4588-8c4f-93964b3bf95c|PROCEDURE:PTS_RULE_ITERATOR|RULEID:BR-TX-ALL-023|ERROR_CODE:0|ERROR_DESC: \r", "@version" => "1", "@timestamp" => "2016-09-15T13:24:13.064Z", "path" => "C:/ELK/Logstash/logstash-2.3.4/pts/pts_log.txt", "host" => "HRODRIGUEZ", "type" => "pts", "l_pts_id1" => "PTS_ID", "pts_id1" => "PI30286", "l_trxId1" => "TRXID", "trxid1" => "9a80fd8e-8946-4588-8c4f-93964b3bf95c", "l_process" => "PROCEDURE", "process" => "PTS_RULE_ITERATOR", "l_processStat" => "RULEID", "processStat" => "BR" } ``` the custom patterns are: ``` _PTS_LOG \[[A-Z]+\_[A-Z]+\]_ _POOL \([a-z]+\-[0-9]\-[a-z]+\-[0-9]\)_ _TRXID [a-z0-9-]+_ _FIELD_NAME [A-Z_\]+_ _RULE_ID [(.\-\d\D)]+_ _ERROR_CODE [(\d\D)]{1,2}_ _ERROR_DESC [(\d\D)]+_ _PTS_ID [A-Z0-9_\]+_ _PROC [A-Z_\]+_ _POST [A-Z_\]+_ _OPERATION [A-Z0-9_\]+_ ``` Finally, this is the filter part detailed: ``` filter { if [type] == "pts" { grok { patterns_dir => ["C:/ELK/Logstash/logstash-2.3.4/pts_patterns"] match => [ "message", "%{FIELD_NAME:l_pts_id1}:%{PTS_ID:pts_id1}\|%{FIELD_NAME:l_trxId1}:%{TRXID:trxid1}\|%{FIELD_NAME:l_process}:%{FIELD_NAME:process}\|%{FIELD_NAME:l_processStat}:%{FIELD_NAME:processStat}", "message", "%{FIELD_NAME:l_pts_id2}:%{PTS_ID:pts_id2}\|%{FIELD_NAME:l_trxId2}:%{TRXID:trxid2}\|%{FIELD_NAME:l_proc}:%{PROC:proc}\|%{FIELD_NAME:l_postmode}:%{POST:postmode}\|%{FIELD_NAME:l_operation}:%{OPERATION:operation}\|%{FIELD_NAME:l_errorcode}:%{ERROR_CODE:errorcode}\|%{FIELD_NAME:l_errordesc}:%{ERROR_DESC:error_desc}", "message", "%{FIELD_NAME:l_pts_id3}:%{PTS_ID:pts_id3}\|%{FIELD_NAME:l_trxId3}:%{TRXID:trxid3}\|%{FIELD_NAME:l_proc2}:%{PROC:proc2}\|%{FIELD_NAME:l_ruleid}:%{RULE_ID:ruleid}\|%{FIELD_NAME:l_errorcode2}:%{ERROR_CODE:errorcode}\|%{FIELD_NAME:l_errordesc}:%{ERROR_DESC:error_desc2}" ] } } } ``` Thanks in advance regard Dario R ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 15, 2016, 6:20pm UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/4 "2016-09-15T18:20:39Z")
So what's missing from the output? ---
**Author:** ![rodher](https://avatars.discourse-cdn.com/v4/letter/r/50afbb/32.png) [@rodher](https://discuss.elastic.co/u/rodher)\ **Post date:** [September 15, 2016, 7:07pm UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/5 "2016-09-15T19:07:16Z")
The file has three kind of lines. Also, the grok filter has 3 messages each with a parser. But the output only has the variables of one of message, that means the variables of the message parser are missing. I don't know why this happen, because in grok debugger all is fine. Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 17, 2016, 5:35pm UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/6 "2016-09-17T17:35:51Z")
What probably happens is that the first grok expression matches all messages so the filter doesn't bother trying with the rest. Either list the expressions in reverse other with the most specific expression first or add a `$` anchor at the end of each expression so that only exact matches are accepted. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:38am UTC](https://discuss.elastic.co/t/parser-for-multiple-line-file/60555/7 "2017-07-06T04:38:06Z")