# \_grokparsefailure using regex filter with multiline codec **URL:** \ **Category:** Logstash\ **Created:** [March 14, 2018, 4:20pm UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949 "2018-03-14T16:20:34Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![martin.g](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@martin.g](https://discuss.elastic.co/u/martin.g)\ **Post date:** [March 14, 2018, 4:20pm UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949/1 "2018-03-14T16:20:34Z")
Hi all, i am trying to parse in Logstash 6.2.2 this log: ``` --- BEGIN ----- 4.3.2018 13:29:26.286 --- Identification: ABC_a1262e23-74a3-4f57-bd79-ee025805bee5 (c0cc8fa7-d8ad-4cda-8a27-c1af03794ca0) Message: Get signed data object ended successfuly. Identification: ABC_a1262e23-74a3-4f57-bd79-ee025805bee5 (c0cc8fa7-d8ad-4cda-8a27-c1af03794ca0) --- END ----- 4.3.2018 13:29:26.286 (2,4811 ms) --- --- BEGIN ----- 4.3.2018 13:29:26.302 --- Identification: ABC_a1262e23-74a3-4f57-bd79-ee025805bee5 (c0cc8fa7-d8ad-4cda-8a27-c1af03794ca0) Message: Get signed data object. Identification: ABC_a1262e23-74a3-4f57-bd79-ee025805bee5 (c0cc8fa7-d8ad-4cda-8a27-c1af03794ca0) --- END ----- 4.3.2018 13:29:26.302 (2,4565 ms) --- --- BEGIN ----- 4.3.2018 13:29:26.302 --- Identification: ABC_a1262e23-74a3-4f57-bd79-ee025805bee5 (c0cc8fa7-d8ad-4cda-8a27-c1af03794ca0) Message: Get signed data object. Identification: ABC_a1262e23-74a3-4f57-bd79-ee025805bee5 (c0cc8fa7-d8ad-4cda-8a27-c1af03794ca0) --- END ----- 4.3.2018 13:29:26.302 (2,4565 ms) --- --- BEGIN ----- 4.3.2018 13:28:09.330 --- Identification: ABC_36db79f6-dc39-4df3-82f3-297d72316bb2 (c846e462-e755-4aae-87cf-89e556a355c2) Message: Initialization started. Message: Initializing config reader. 95,4582 ms Message: Initializing error log. 0,0997 ms Message: Initializing plugins: 6,1417 ms - 2 item(s) - msg from LoadPlugins: LoadPlugins Search files "XYZ.*.dll": 0,2736 ms - 2 file(s). CurrentDomainAssemblies: 0,1078 ms - count 156. Processing file: XYZ.Pdf.dll. CheckToken for PdfPlugin: 0,0228 ms. Processing file end: 1,6042 ms. Processing file: XYZ.Png.dll. CheckToken for PngPlugin: 0,0213 ms. Processing file end: 1,2178 ms. GetPlugins Foreach gettypes: 0,0313 ms. Foreach types: 0,2379 ms. GetPlugins end: 0,2882 ms. LoadPlugins end: 6,1281 ms. Message: Create xml document: 5,6369 ms Message: Create namespace manager: 0,0076 ms Message: Check signature version: 0,0203 ms Message: Verify signature schema 6,367 ms - removed node count: 0 item(s). Message: validateSignaturePolicy: ASDSDFGDSFGSDFGDSFG Message: Initialization ended successfuly. Identification: ABC_36db79f6-dc39-4df3-82f3-297d72316bb2 (c846e462-e755-4aae-87cf-89e556a355c2) --- END ----- 4.3.2018 13:28:09.642 (321,7862 ms) --- ``` In my conf i am at begining trying to parse start datetime - --- BEGIN ----- **4.3.2018 13:28:09.330** --- My conf is: ``` input { file { path => ["file.log"] start_position => "beginning" sincedb_path => "/dev/null" codec => multiline { pattern => "^--- BEGIN" negate => true what => "previous" auto_flush_interval => 30 } } } filter { grok { match => ["message", "(?(?<=--- BEGIN ----- ).*(?= ---))"] } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["localhost:9200"] action => "index" index => "log_index" } stdout { } } ``` Parsing multiline events working fine but parsing start is returning **\_grokparsefailure** , even my grok is working in grokdebug [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/). Interesting is that this grok is working fine ``` grok { match => ["message", "(?(?<=--- BEGIN ----- ).{21})"] } ``` but my start string is not having fixed length. Do you have any idea where is the problem? Regards Martin ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 14, 2018, 4:41pm UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949/2 "2018-03-14T16:41:48Z")
> [@martin.g](#): > > Do you have any idea where is the problem? No, but isn't ```auto match => ["message", "--- BEGIN ----- (?[ 0-9:\.]+) " ] ``` a simpler (and cheaper) solution? ---
**Author:** ![martin.g](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@martin.g](https://discuss.elastic.co/u/martin.g)\ **Post date:** [March 15, 2018, 8:12am UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949/3 "2018-03-15T08:12:07Z")
It works fine and I find your solution as better then mine. Thank you very much. ---
**Author:** ![martin.g](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@martin.g](https://discuss.elastic.co/u/martin.g)\ **Post date:** [March 20, 2018, 11:58am UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949/4 "2018-03-20T11:58:32Z")
``` "--- BEGIN ----- (?.*) ---" "--- END ----- (?.*) \(" "--- END ----- (?.*) \((?.*) ms\)" "Identification: (?ABC_.*) \((?.*)\)" ``` _ **Start** _ is not working, _ **end** _ separately works fine, _ **end** _ in combination with _ **duration** _ is OK as well but _ **duration** _ not. _ **identification1** _ and _ **identification2** _ is not working. All of them work in grokdebug. It looks like logstash has problem with .\* in some cases, and i think it is bug. So I am using this but i think it is work around: ``` "--- BEGIN ----- (?[0-9:\.]+) ---" "--- END ----- (?[0-9:\.]+) \((?[0-9,]+) ms\)" "Identification: (?ABC_[a-z0-9-]+) \((?[a-z0-9-]+)\)" ``` I have another question, how can I get array **messages** with all values after "Meaasage: " till end of line as array items. Thank you very much Martin ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 20, 2018, 2:07pm UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949/5 "2018-03-20T14:07:36Z")
> [@martin.g](#): > > how can I get array messages with all values after "Meaasage: " till end of line as array items Interesting problem! ```auto filter { grok { # Get every occurrence of tab followed by Message: until a line that has tab followed by not-M match => [ "message", "(? Message:.* ) [^M]" ] } mutate { gsub => [ # Remove the tab+Message: "messages", " Message: ", "", # Delete tab, then one or more not newline followed by a newline "messages", " [^ ]+ ", "" ] } mutate { # Split on newline split => { "messages" => " " } } } ``` Will get you ```auto "messages" => [ [0] "Initialization started.", [1] "Initializing config reader. 95,4582 ms", [2] "Initializing error log. 0,0997 ms", [3] "Initializing plugins: 6,1417 ms - 2 item(s) - msg from LoadPlugins: ", [4] "Create xml document: 5,6369 ms", [5] "Create namespace manager: 0,0076 ms", [6] "Check signature version: 0,0203 ms", [7] "Verify signature schema 6,367 ms - removed node count: 0 item(s).", [8] "validateSignaturePolicy: ASDSDFGDSFGSDFGDSFG", [9] "Initialization ended successfuly." ] ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 17, 2018, 2:07pm UTC](https://discuss.elastic.co/t/grokparsefailure-using-regex-filter-with-multiline-codec/123949/6 "2018-04-17T14:07:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.