# Extract on various lines with logstash

**URL:** <https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916>\
**Category:** Logstash\
**Created:** [March 31, 2016, 10:27am UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916 "2016-03-31T10:27:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![oth1113](https://avatars.discourse-cdn.com/v4/letter/o/13edae/32.png) [@oth1113](https://discuss.elastic.co/u/oth1113)\
**Post date:** [March 31, 2016, 10:27am UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/1 "2016-03-31T10:27:09Z")

</div>

so I just discovered logstash and I managed to extract data from log files. This time, I have to extract information on MULTIlines, I show you an example:

`2016-03-07 14:09:11,613 INFO [][com.ole.ecom.jms.crm.JmsCrmSender] Envoi du message ...<?xml version="1.0" encoding="UTF-8" standalone="yes"?>`

`<root xmlns="http://wlsosb.gjkjdhdfhkllmld.crm/"> <entry>`  
`<clients> <code>8800356499460</code> <raisonsociale>Ole</raisonsociale> <siret>51770313800088</siret> <civilite>M.</civilite> <nom>Aurius</nom> <prenom>Gerard</prenom> <telephone>0614666644</telephone> <email>outiztest+prolivraison@gmail.com</email> </clients>`

I have to recover, information on the first line and I must also be able to extract the number in between `<code> 8800356499460 </ code>` and email between the `<email> outiztest+prolivraison@gmail.com </ email >`  
I created a file conf but it doesnt work , i want to know how can i do a file conf to this type of file (.log) that contains informations on many lines.

if you see how I can do, help me I would be very grateful

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 31, 2016, 10:37am UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/2 "2016-03-31T10:37:28Z")

</div>

Use a multiline codec to join the lines of a multiline event to a single Logstash event, then use the grok filter to extract the various pieces of the log into separate fields, then use the xml filter to parse the XML document.

---

<div class="post-metadata">

**Author:** ![oth1113](https://avatars.discourse-cdn.com/v4/letter/o/13edae/32.png) [@oth1113](https://discuss.elastic.co/u/oth1113)\
**Post date:** [March 31, 2016, 11:02am UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/3 "2016-03-31T11:02:00Z")

</div>

Myfile conf is a s flow : my grok filters works , but i dont know why my file don't work

input {  
file {  
type =\> "HYBRIS"  
path =\> "C:\Users\O6665391\ELK\logstash-2.2.2\multiline.log"  
codec =\> multiline {  
pattern =\> "^%{TIMESTAMP\_ISO8601}"  
negate =\> true  
what =\> "previous"  
}

}  
}

filter {  
if [type] == "HYBRIS" {  
grok {  
match =\> { "message" =\> "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s\*)[][%{NOTSPACE:CLASS}]( ## )(debut d'envoie de)(\s\*)[%{INT:DebENV}]( message pour le flux )%{WORD:FLUX}" }

```
	  add_field => {
						"haspath" => "yes"
						"endpoint" => "%{FLUX}"
				}
				break_on_match => true
				add_tag => "msgFlux"
				remove_tag => ["_grokparsefailure"]
	}
	
	grok {
	  match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)\[\]\[%{NOTSPACE:CLASS}\]((.*)(\n))*(\s*)<order>(\n)(\s*)(<numCommande>)%{GREEDYDATA:NUMCMD}(</numCommande>)" }
	 
	  add_field => {
						"haspath" => "yes"
						"endpoint" => "%{NUMCMD}"
				}
				break_on_matchl => true
				add_tag => "orderCRM"
				remove_tag => ["_grokparsefailure"]
	}
	
	grok {
	  match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)\[\]\[%{NOTSPACE:CLASS}\](.*)(\n)(.*)(\n)(.*)(\n)(\s*)<clients>((.*)(\n))*(\s*)(<email>)%{GREEDYDATA:EMAIL}(</email>)" }
	 
	  add_field => {
						"haspath" => "yes"
						"endpoint" => "%{EMAIL}"
				}
				break_on_match => true
				add_tag => "clientCRM"
				remove_tag => ["_grokparsefailure"]
	}

```

}

date {  
match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]  
}

if "orderCRM" not in [tags] and "clientCRM" not in [tags] and "msgFlux" not in [tags] {  
drop { }  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:8080"]  
}  
stdout { codec =\> rubydebug }

}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 31, 2016, 11:06am UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/4 "2016-03-31T11:06:02Z")

</div>

What do you mean by "won't work"? Please give a complete example of input, current output, and expected result.

---

<div class="post-metadata">

**Author:** ![oth1113](https://avatars.discourse-cdn.com/v4/letter/o/13edae/32.png) [@oth1113](https://discuss.elastic.co/u/oth1113)\
**Post date:** [March 31, 2016, 12:23pm UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/5 "2016-03-31T12:23:14Z")

</div>

I show you an example line :  
`2016-03-07 14:09:31,607 INFO [][com.outiz.ecom.jms.crm.JmsCrmSender] Envoi du message ...<?xml version="1.0" encoding="UTF-8" standalone="yes"?>`  
`<root xmlns="http://wlsosb.pointp.saint-gobain.net/hybris_crm/"> <order> <numCommande>00130175</numCommande> <idHybris>8804793647149</idHybris> <numCompteHybris>8803711877124</numCompteHybris> <commandePasseParIdUser></commandePasseParIdUser> <commandePasseParProfil>WEB_PRO</commandePasseParProfil> <produitsAsiles></produitsAsiles> <montantTTC>222.7</montantTTC> <montantHT>192.09</montantHT> <dateCommande>2016-03-07T11:26:16.000+01:00</dateCommande> <horodatageCreation>2016-03-07T11:26:16.000+01:00</horodatageCreation> <horodatageModification>2016-03-07T14:09:24.803+01:00</horodatageModification> <contact>outiztest+wuwu@gmail.com</contact> <coupon>true</coupon> <codeCoupon>COUPON MAI</codeCoupon> <codePromotion>199</codePromotion> <nomCoupon>1000 utilisation</nomCoupon> <reductionCouponTTC>0.0</reductionCouponTTC> <reductionCouponHT>0.0</reductionCouponHT> <reductionCouponHTPercent>0.00</reductionCouponHTPercent> <montantBundle>0</montantBundle> <montantReducBundle></montantReducBundle> <montantReducBundleTTC></montantReducBundleTTC> <lieuPayment>WEB</lieuPayment> <moyenPayment>VISA</moyenPayment> <canalVente>OUTIZ</canalVente>`

and I want to get the first line with date ..., and the commande number on this example.  
I write my filtre for this type of message as follows :  
`grok { match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)\[\]\[%{NOTSPACE:CLASS}\]((.*)(\n))*(\s*)<order>(\n)(\s*)(<numCommande>)%{GREEDYDATA:NUMCMD}(</numCommande>)" }`  
`add_field => { "haspath" => "yes" "endpoint" => "%{NUMCMD}" } break_on_matchl => true add_tag => "orderCRM" remove_tag => ["_grokparsefailure"] }`

and input as follows :  
`input { file { type => "HYBRIS" path => "C:\Users\O6665391\ELK\logstash-2.2.2\multiline.log" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" }}}`

the problem is that on grokdebug, my filter works but with the .conf file, it doesn't work.  
I have no error message, but it does not work

---

<div class="post-metadata">

**Author:** ![oth1113](https://avatars.discourse-cdn.com/v4/letter/o/13edae/32.png) [@oth1113](https://discuss.elastic.co/u/oth1113)\
**Post date:** [March 31, 2016, 1:54pm UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/6 "2016-03-31T13:54:53Z")

</div>

he apparently there's no one to help me .  
thank you:sweat:

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 31, 2016, 2:01pm UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/7 "2016-03-31T14:01:48Z")

</div>

Questions asked here are typically answered by volunteers and you should set your expectations accordingly.

I don't have time to help further in this thread. Maybe someone else can help out.

---

<div class="post-metadata">

**Author:** ![oth1113](https://avatars.discourse-cdn.com/v4/letter/o/13edae/32.png) [@oth1113](https://discuss.elastic.co/u/oth1113)\
**Post date:** [March 31, 2016, 2:22pm UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/8 "2016-03-31T14:22:08Z")

</div>

okay, it 's okay .  
I contiuner my research on my side .

thanks anyway

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:04am UTC](https://discuss.elastic.co/t/extract-on-various-lines-with-logstash/45916/9 "2017-07-06T05:04:29Z")

</div>