# Reading over multiple lines with grok **URL:** \ **Category:** Logstash\ **Created:** [August 5, 2019, 12:05pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786 "2019-08-05T12:05:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Kapi](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@Kapi](https://discuss.elastic.co/u/Kapi)\ **Post date:** [August 5, 2019, 12:06pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786/1 "2019-08-05T12:06:00Z")
Hi I have this Log where I want to match the start of a thread of device with the corresponding spoolid. ``` 30 Jul 2019 09:56:28 DEVICE0 START_THREAD THREAD_ID(B01234) 30 Jul 2019 09:56:28 DEVICE1 START_THREAD THREAD_ID(B12345) 30 Jul 2019 09:56:28 DEVICE2 START_THREAD THREAD_ID(A12345) 30 Jul 2019 09:56:28 DEVICE1 PROCESSING SPOOLID 100 30 Jul 2019 09:56:28 DEVICE2 PROCESSING SPOOLID 101 30 Jul 2019 09:56:28 DEVICE2 PROCESSING SPOOLID 101 30 Jul 2019 09:56:28 DEVICE1 PROCESSING SPOOLID 100 30 Jul 2019 09:56:28 DEVICE1 SPOOLID 100 PROCESSED 30 Jul 2019 09:56:28 DEVICE2 SPOOLID 101 PROCESSED 30 Jul 2019 09:56:28 DEVICE2 STOP_THREAD THREAD_ID(B12345) 30 Jul 2019 09:56:28 DEVICE2 STOP_THREAD THREAD_ID(A12345) 30 Jul 2019 09:56:28 DEVICE2 START_THREAD THREAD_ID(A23456) 30 Jul 2019 09:56:29 DEVICE2 PROCESSING SPOOLID 102 30 Jul 2019 09:56:29 DEVICE2 PROCESSING SPOOLID 102 30 Jul 2019 09:56:29 DEVICE2 SPOOLID 102 PROCESSED 30 Jul 2019 09:56:29 DEVICE2 STOP_THREAD THREAD_ID(A23456) 30 Jul 2019 09:56:29 DEVICE0 PROCESSING SPOOLID 99 30 Jul 2019 09:56:29 DEVICE0 SPOOLID 99 PROCESSED 30 Jul 2019 09:56:29 DEVICE0 STOP_THREAD THREAD_ID(B12345) ``` To get the corresponding fields I have this code: `(?m)%{DATA}\>\s+(?\b%{DATA}\b)\s*START_THREAD%{DATA}\k\s+SPOOLID\s+%{NUMBER:num}\s+PROCESSED` In the Heroku Grok Debugger I get the correct fields but not in my Logstash on my server because there I get line by line and therefore a \_grokparsefailure error. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 5, 2019, 12:20pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786/2 "2019-08-05T12:20:54Z")
I would use an aggregate filter for that... ``` dissect { mapping => { "message" => "%{[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} <%{id}> %{dev} %{restOfLine}" } } date { match => ["[@metadata][ts]", "dd MMM YYYY HH:mm:ss" ] } grok { match => { "restOfLine" => [ "^START_THREAD THREAD_ID\(%{DATA:thread}\)", "^STOP_THREAD THREAD_ID\(%{DATA:thread}\)", "^PROCESSING SPOOLID %{NUMBER:spool}", "^SPOOLID %{NUMBER:spool} PROCESSED" ] } } if " START_THREAD " in [message] { aggregate { task_id => "%{dev}" code => 'map["thread"] = event.get("thread")' map_action => "create" } } if " PROCESSED" in [message] { aggregate { task_id => "%{dev}" code => "event.set('thread', map['thread'])" map_action => "update" end_of_task => true timeout => 120 } } else { drop {} } ``` ---
**Author:** ![Kapi](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@Kapi](https://discuss.elastic.co/u/Kapi)\ **Post date:** [August 5, 2019, 2:21pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786/3 "2019-08-05T14:21:51Z")
Thank you for your reply. That is also a way, but it is still not clear for me why the (?m) in my grok code is not recognized. Is there a special configuration needed in logstash.yml for this to work? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 5, 2019, 2:55pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786/4 "2019-08-05T14:55:17Z")
I don't think you can do it in grok. If you use a multiline codec on the input to ingest the complete set of lines as one event then the grok pattern matches, but only once ``` "message" => "30 Jul 2019 09:56:28 DEVICE0 START_THREAD THREAD_ID(B01234)\n30 Jul 2019 09:56:28 DEVICE1 START_THREAD THREAD_ID(B12345)\n30 Jul 2019 09:56:28 DEVICE2 START_THREAD THREAD_ID(A12345)\n30 Jul 2019 09:56:28 DEVICE1 PROCESSING SPOOLID 100 \n30 Jul 2019 09:56:28 DEVICE2 PROCESSING SPOOLID 101\n30 Jul 2019 09:56:28 DEVICE2 PROCESSING SPOOLID 101\n30 Jul 2019 09:56:28 DEVICE1 PROCESSING SPOOLID 100\n30 Jul 2019 09:56:28 DEVICE1 SPOOLID 100 PROCESSED\n30 Jul 2019 09:56:28 DEVICE2 SPOOLID 101 PROCESSED\n30 Jul 2019 09:56:28 DEVICE2 STOP_THREAD THREAD_ID(B12345) \n30 Jul 2019 09:56:28 DEVICE2 STOP_THREAD THREAD_ID(A12345)\n30 Jul 2019 09:56:28 DEVICE2 START_THREAD THREAD_ID(A23456)\n30 Jul 2019 09:56:29 DEVICE2 PROCESSING SPOOLID 102\n30 Jul 2019 09:56:29 DEVICE2 PROCESSING SPOOLID 102\n30 Jul 2019 09:56:29 DEVICE2 SPOOLID 102 PROCESSED\n30 Jul 2019 09:56:29 DEVICE2 STOP_THREAD THREAD_ID(A23456) \n30 Jul 2019 09:56:29 DEVICE0 PROCESSING SPOOLID 99 \n30 Jul 2019 09:56:29 DEVICE0 SPOOLID 99 PROCESSED\n30 Jul 2019 09:56:29 DEVICE0 STOP_THREAD THREAD_ID(B12345)", "device" => "DEVICE0", "num" => "99" ``` ---
**Author:** ![Kapi](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@Kapi](https://discuss.elastic.co/u/Kapi)\ **Post date:** [August 7, 2019, 2:47pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786/5 "2019-08-07T14:47:47Z")
I need the device and the spoolid for the task\_id, because the different spoolid for the same device can be mixed even before one device has finished processing . Howerver when I do it like this I do not get the thread number which I also need. Is there a solution? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 4, 2019, 2:47pm UTC](https://discuss.elastic.co/t/reading-over-multiple-lines-with-grok/193786/6 "2019-09-04T14:47:50Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.