# Extract multiline log with not id field present **URL:** \ **Category:** Logstash\ **Created:** [March 1, 2021, 6:58pm UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848 "2021-03-01T18:58:40Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![E\_T\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e_t_n/32/77872_2.png) [@E\_T\_N](https://discuss.elastic.co/u/E_T_N)\ **Post date:** [March 1, 2021, 6:58pm UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/1 "2021-03-01T18:58:40Z")
Hello. I need to build a record composed of multiple lines where there is no clear identifier - There is a word that tells me where the task starts in this case "select" - There is a word that tells me where the task ends in this case "end log" - I don´t have a field that allows me to uniquely identify - Important information exists on various lines - everything else must be ignored I have a log file with similar estructure: ```auto 05 Feb 2021 14:00:00,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:00:00,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:00:00,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:00:01,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:00:01,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:00:01,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:00:02,213 [AAA-11] INFO - select name 05 Feb 2021 14:00:02,213 [AAA-11] INFO - Include important info 05 Feb 2021 14:00:02,231 [AAA-11] INFO - message Value 1 Value 2 05 Feb 2021 14:00:03,131 [AAA-11] INFO - Include other important info 05 Feb 2021 14:00:03,131 [AAA-11] INFO - end log 05 Feb 2021 14:00:03,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:00:03,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:00:03,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:00:04,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:00:04,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:00:04,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:00:05,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:00:05,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:00:05,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:01:00,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:01:00,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:01:00,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:01:01,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:01:01,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:01:01,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:01:02,213 [AAA-11] INFO - select name 05 Feb 2021 14:01:02,213 [AAA-11] INFO - Include inportant inf 05 Feb 2021 14:01:02,231 [AAA-11] INFO - message Value 1 Value 2 05 Feb 2021 14:01:03,131 [AAA-11] INFO - Include other important info 05 Feb 2021 14:01:03,131 [AAA-11] INFO - end log 05 Feb 2021 14:01:03,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:01:03,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:01:03,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:01:04,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:01:04,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:01:04,231 [AAA-11] INFO - Line3[: 0] 05 Feb 2021 14:01:05,213 [AAA-11] INFO - Line A:[0] 05 Feb 2021 14:01:05,231 [AAA-11] INFO - Line2:[0] 05 Feb 2021 14:01:05,231 [AAA-11] INFO - Line3[: 0] ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 1, 2021, 8:05pm UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/2 "2021-03-01T20:05:33Z")
If you are using a file input you could use a multiline codec to roll up lines until an "end log" is seen. You could then use mutate+gsub to remove everything before "select". Then pick the resulting event apart with grok, perhaps. ---
**Author:** ![E\_T\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e_t_n/32/77872_2.png) [@E\_T\_N](https://discuss.elastic.co/u/E_T_N)\ **Post date:** [March 1, 2021, 9:24pm UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/3 "2021-03-01T21:24:54Z")
Logstash insert documents with tag: ``` `"multiline_codec_max_lines_reached"` ``` ```auto input { file { path => ["./my.log"] start_position => "beginning" sincedb_path => "NUL" codec => multiline { pattern => "^end log" what => "previous" negate => true } } } filter { mutate { gsub => ["message", '\r', ""] } } output { elasticsearch { hosts => ["http://127.0.0.1:9200"] index => "send" document_type => "_doc" } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 1, 2021, 9:50pm UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/4 "2021-03-01T21:50:05Z")
> [@E\_T\_N](#): > > `pattern => "^end log"` That is anchored to start of line. If end log is not at the start of the line that will roll up the entire file into a single event. ---
**Author:** ![E\_T\_N](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e_t_n/32/77872_2.png) [@E\_T\_N](https://discuss.elastic.co/u/E_T_N)\ **Post date:** [March 1, 2021, 10:14pm UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/5 "2021-03-01T22:14:52Z")
For this case the word "select" init the event and the "end log" in other line finish the event. I would have to capture the lines that are in between. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 2, 2021, 12:11am UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/6 "2021-03-02T00:11:17Z")
Remove the ^ from your pattern option. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 30, 2021, 12:12am UTC](https://discuss.elastic.co/t/extract-multiline-log-with-not-id-field-present/265848/7 "2021-03-30T00:12:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.