# Log stash filter for multi line occurrences and capture in custom field **URL:** \ **Category:** Logstash\ **Created:** [February 8, 2019, 1:26pm UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633 "2019-02-08T13:26:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Post date:** [February 8, 2019, 1:26pm UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633/1 "2019-02-08T13:26:57Z")
We are having a huge log file in which we are applying many filter and storing it as a specific filed in that below line has, more than one occurance(3 to 5 times), In that we wanted to capture last occurance under filed name called "build\_Status". But for us either we are getting nil value for build\_Status or getting lots of lines next to that line **Current state: job state = 'Succeded'**. Please review below and suggest us **Log File example:** [2019-02-04 07:16:46Z INFO StepsRunner] Current state: job state = 'Succeeded' Some Additional lines [2019-02-04 07:26:46Z INFO StepsRunner] Current state: job state = 'Failed' Some Additional lines [2019-02-04 07:28:46Z INFO StepsRunner] Current state: job state = 'Canceled' Some Additional lines [2019-02-04 07:32:46Z INFO StepsRunner] Current state: job state = 'Failed' Some Additional lines **Logstash Config file:** input { beats { client\_inactivity\_timeout =\> 1200 port =\> 5002 } } filter { if [message] =~ "Version: 2.122.1" { grok { add\_tag =\> ["start"] match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:build\_StartTime}" } } } grok { add\_tag =\> ["start"] match =\> { "message" =\> "(?\ ERR .\*)" } } if [message] =~ "Current state: job state" { grok { add\_tag =\> ["start"] match =\> { "message" =\> "Current state: job state = '(?\.+)'" } } if ![build\_Status] or [build\_Status] == 'nil' { mutate { add\_tag =\> ["start"] add\_field =\> {"build\_Status" =\> "Succeded"} } } } if [message] =~ "Job completed." { grok { add\_tag =\> ["start"] match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:build\_EndTime}" } } } grok { add\_tag =\> ["end"] match =\> { "message" =\> "Job completed." } } if "start" in [tags] { aggregate { task\_id =\> "%{source}" code =\> " map['build\_Status'] = event.get('build\_Status') unless event.get('build\_Status').nil? map['build\_ErrorMessage'] = event.get('build\_ErrorMessage') unless event.get('build\_ErrorMessage').nil? map['build\_StartTime'] = event.get('build\_StartTime') unless event.get('build\_StartTime').nil? map['build\_EndTime'] = event.get('build\_EndTime') unless event.get('build\_EndTime').nil? " } } if "end" in [tags] { aggregate { task\_id =\> "%{source}" code =\> " event.set('build\_ErrorMessage', map['build\_ErrorMessage']) event.set('build\_Status', map['build\_Status']) event.set('build\_StartTime', map['build\_StartTime']) event.set('build\_EndTime', map['build\_EndTime']) " end\_of\_task =\> true } } if "end" not in [tags] { drop { } } mutate { remove\_field =\> ["message"] } mutate { remove\_tag =\> ["start"] } } output { elasticsearch { hosts =\> ["elasticsearch:9200"] index =\> "tfslog-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } **FileBeat Config:** ## filebeat: prospectors: paths: - C:/\*\*\*\*\*\*\*\*\*\*\* _/Worker\__.log multiline: pattern: '^[.+]' negate: true match: after max\_line: 999 timeout: 60 output.logstash: hosts: ["Logstash\_Server:5002"] **OutPut:** ""build\_StartTime" =\> "2019-02-05 07:50:03Z" "build\_BuildId" =\> "1448697", "build\_ErrorMessage" =\> nil, "beat" =\> { "hostname" =\> "########", "version" =\> "6.3.0", "name" =\> "#######" }, "build\_Status" =\> nil, "build\_EndTime" =\> "2019-02-05 09:40:03Z" **Some Times build\_Status contains:** "build\_Status" =\> "'\n[2019-02-05 07:50:32Z INFO JobRun ner] Job result after all pre-job steps finish: \n[2019-02-05 07:50:32Z INFO Job Runner] Run all job steps.\n[2019-02-05 07:50:32Z INFO StepsRunner] Processing s tep: DisplayName='$/Roche.DP.NewGen/BuildTools/Dev/Cake/build.cmd', ContinueOnEr ror=False, Enabled=True\n[2019-02-05 07:50:32Z INFO ExpressionManager] Evaluatin g: succeeded()\n[2019-02-05 07:50:32Z INFO ExpressionManager] Expanded: True\n[2 019-02-05 07:50:32Z INFO ExpressionManager] Result: True\n[2019-02-05 07:50:32Z **etc** ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 8, 2019, 2:07pm UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633/2 "2019-02-08T14:07:39Z")
That configuration never sets build\_DefinitionName, so it will drop all events. You are showing us one configuration and asking us to help fix the behaviour of a different configuration. We cannot do that. ---
**Author:** ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Post date:** [February 11, 2019, 5:30am UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633/3 "2019-02-11T05:30:58Z")
Thanks for the reply. I used **or** condition so if build\_DefinitionName is there then it will drop else it will ignore. But as of now I removed the build\_DefinitionName part from above config. Now please have a look and help us. We need to Capture build\_Status = Failed or build\_Status = Succeed or build\_Status = Canceled from the below log line, Which has multiple occurrences and we wanted to capture the final occurrence [2019-02-04 07:16:46Z INFO StepsRunner] Current state: job state = 'Succeeded' ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 11, 2019, 1:03pm UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633/4 "2019-02-11T13:03:45Z")
You are doing an aggregate with a non-existent field as task\_id. That does not work. ---
**Author:** ![vasudevan](https://avatars.discourse-cdn.com/v4/letter/v/f07891/32.png) [@vasudevan](https://discuss.elastic.co/u/vasudevan)\ **Post date:** [February 12, 2019, 5:35am UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633/5 "2019-02-12T05:35:49Z")
No It's working for us. I guess you didn't get our issue or I am not explaining it properly anyhow we can close this ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 12, 2019, 5:35am UTC](https://discuss.elastic.co/t/log-stash-filter-for-multi-line-occurrences-and-capture-in-custom-field/167633/6 "2019-03-12T05:35:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.