# Grok : Extract multiple lines format occurring between two lines of log data

**URL:** https://discuss.elastic.co/t/grok-extract-multiple-lines-format-occurring-between-two-lines-of-log-data/148962
**Category:** Logstash
**Created:** [September 18, 2018, 10:49am UTC](https://discuss.elastic.co/t/grok-extract-multiple-lines-format-occurring-between-two-lines-of-log-data/148962 "2018-09-18T10:49:47Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![richagautam](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@richagautam](https://discuss.elastic.co/u/richagautam)
#### Post date: [September 18, 2018, 10:49am UTC](https://discuss.elastic.co/t/grok-extract-multiple-lines-format-occurring-between-two-lines-of-log-data/148962/1 "2018-09-18T10:49:48Z")

</div>

Hi Team,

I am trying to parse multiple log files where log syntax are not same and extract differnt lines which is coming between 2 lines of log data. Below is the sample data

Feb 12 18:24:26 localhost.localdomain: TC START SampleTC()"  
Feb 10 18:24:34 localhost.localdomain kibana[697]: WARNING in TestCase2()  
Feb 10 18:24:40 localhost.localdomain kibana[697]: FAILED in TestCase2()  
Feb 10 18:24:42 localhost.localdomain kibana[697]: FAILED String1 Strng2 in TestCase2()  
Feb 12 18:26:24 localhost.localdomain: TC COMPLETED SampleTC()

I have to extract all the failure message ( like warning , fail, error) between SAMPLE Start and SAMPLE complete lines which is coming the log file and maintain the order of data.

Currently I am able to extract releavent log lines by using multiple grok.  
To implement the condition i am trying to implement below filter but i am not sure whether this approach is correct. Also I want to add TCNAME which i added in first grok in other grok also, but i am not able to achieve this.

filter {  
grok  
{  
match =\> {"message" =\> "%{SYSLOGTIMESTAMP:Time} %{GREEDYDATA:host} (?%{WORD})[[0-9]\*]: TC START %{GREEDYDATA:DataString1}"}  
add\_field =\> { "[TCNAME]" =\> "%{DataString1}" }  
remove\_tag =\> ["\_grokparsefailure"]

```
              }
			  
			  if [message] != "TESTCASE COMPLETED %{DataString}"
					  
					  {
						  
						if "_grokparsefailure" in [tags] {
							grok
							 {
								match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{HOSTNAME:HOST} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: (?<DataString>WARNING %{GREEDYDATA})"}
								add_field => { "[TCNAME]" => "%{DataString1}" }
								remove_tag => ["_grokparsefailure"]
							 }
						}		  
						 if "_grokparsefailure" in [tags] {
							grok
							 {
								match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{HOSTNAME:HOST} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: (?<DataString>FAILED %{GREEDYDATA})"}
					  
								add_field => { "[TCNAME]" => "%{DataString1}" }
								remove_tag => ["_grokparsefailure"]									

							 }
						}	
				       }
		 	
			if "_grokparsefailure" in [tags] {
			grok
			 {
				 match => {"message" => "%{SYSLOGTIMESTAMP:Time} %{HOSTNAME:HOST} (?<LOGINTYPE>%{WORD})\[[0-9]*\]\: TC COMPLETED %{GREEDYDATA:DataString}"}
					 
				add_field => { "[TCNAME]" => "%{DataString}"}
				remove_tag => ["_grokparsefailure"]
				
			 }
			}		

```

if "\_grokparsefailure" in [tags]  
{  
drop{}  
}

mutate{  
remove\_field =\> ["@version", "host", "tags", "offset", "type", "input\_type", "errordata"]  
}  
}

---

<div class="post-metadata">

### Author: ![ramindia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramindia/32/32459_2.png) [@ramindia](https://discuss.elastic.co/u/ramindia)
#### Post date: [September 21, 2018, 9:21pm UTC](https://discuss.elastic.co/t/grok-extract-multiple-lines-format-occurring-between-two-lines-of-log-data/148962/2 "2018-09-21T21:21:04Z")

</div>

i did some work same  
have a look this thread

> [@How many matches, we can have in grok filter - as example below](https://discuss.elastic.co/t/how-many-matches-we-can-have-in-grok-filter-as-example-below/148415):
>
> I have assorted logs coming to syslog file as one all.log, i am looking sort using different grok match, is this possible as below : match ? input { file { path =\> "/var/syslog-ng/raw/\*.log" start\_position =\> "beginning" type =\> "logstash-syslog" tags =\> ["logstash-syslog"] } } filter { grok { match =\> { [ "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{HOSTNAME:device\_src} %{NUMBER} %{SYSLOGTIMESTAMP} %{WORD}: %{GREEDYDATA:syslog\_message}", …

is this what you looking ?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 19, 2018, 9:21pm UTC](https://discuss.elastic.co/t/grok-extract-multiple-lines-format-occurring-between-two-lines-of-log-data/148962/3 "2018-10-19T21:21:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.