Extract on various lines with logstash

Elastic Stack Logstash
1

so I just discovered logstash and I managed to extract data from log files. This time, I have to extract information on MULTIlines, I show you an example:

2016-03-07 14:09:11,613 INFO [][com.ole.ecom.jms.crm.JmsCrmSender] Envoi du message ...<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<root xmlns="http://wlsosb.gjkjdhdfhkllmld.crm/"> <entry>
<clients> <code>8800356499460</code> <raisonsociale>Ole</raisonsociale> <siret>51770313800088</siret> <civilite>M.</civilite> <nom>Aurius</nom> <prenom>Gerard</prenom> <telephone>0614666644</telephone> <email>outiztest+prolivraison@gmail.com</email> </clients>

I have to recover, information on the first line and I must also be able to extract the number in between <code> 8800356499460 </ code> and email between the <email> outiztest+prolivraison@gmail.com </ email >
I created a file conf but it doesnt work , i want to know how can i do a file conf to this type of file (.log) that contains informations on many lines.

if you see how I can do, help me I would be very grateful

2

Use a multiline codec to join the lines of a multiline event to a single Logstash event, then use the grok filter to extract the various pieces of the log into separate fields, then use the xml filter to parse the XML document.

3

Myfile conf is a s flow : my grok filters works , but i dont know why my file don't work

input {
file {
type => "HYBRIS"
path => "C:\Users\O6665391\ELK\logstash-2.2.2\multiline.log"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => "previous"
}

}
}

filter {
if [type] == "HYBRIS" {
grok {
match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[ ]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)[][%{NOTSPACE:CLASS}]( ## )(debut d'envoie de)(\s*)[%{INT:DebENV}]( message pour le flux )%{WORD:FLUX}" }

	  add_field => {
						"haspath" => "yes"
						"endpoint" => "%{FLUX}"
				}
				break_on_match => true
				add_tag => "msgFlux"
				remove_tag => ["_grokparsefailure"]
	}
	
	grok {
	  match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[ ]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)\[\]\[%{NOTSPACE:CLASS}\]((.*)(\n))*(\s*)<order>(\n)(\s*)(<numCommande>)%{GREEDYDATA:NUMCMD}(</numCommande>)" }
	 
	  add_field => {
						"haspath" => "yes"
						"endpoint" => "%{NUMCMD}"
				}
				break_on_matchl => true
				add_tag => "orderCRM"
				remove_tag => ["_grokparsefailure"]
	}
	
	grok {
	  match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[ ]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)\[\]\[%{NOTSPACE:CLASS}\](.*)(\n)(.*)(\n)(.*)(\n)(\s*)<clients>((.*)(\n))*(\s*)(<email>)%{GREEDYDATA:EMAIL}(</email>)" }
	 
	  add_field => {
						"haspath" => "yes"
						"endpoint" => "%{EMAIL}"
				}
				break_on_match => true
				add_tag => "clientCRM"
				remove_tag => ["_grokparsefailure"]
	}

}

date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}

if "orderCRM" not in [tags] and "clientCRM" not in [tags] and "msgFlux" not in [tags] {
drop { }
}
}

output {
elasticsearch {
hosts => ["localhost:8080"]
}
stdout { codec => rubydebug }

}

4

What do you mean by "won't work"? Please give a complete example of input, current output, and expected result.

5

I show you an example line :
2016-03-07 14:09:31,607 INFO [][com.outiz.ecom.jms.crm.JmsCrmSender] Envoi du message ...<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<root xmlns="http://wlsosb.pointp.saint-gobain.net/hybris_crm/"> <order> <numCommande>00130175</numCommande> <idHybris>8804793647149</idHybris> <numCompteHybris>8803711877124</numCompteHybris> <commandePasseParIdUser></commandePasseParIdUser> <commandePasseParProfil>WEB_PRO</commandePasseParProfil> <produitsAsiles></produitsAsiles> <montantTTC>222.7</montantTTC> <montantHT>192.09</montantHT> <dateCommande>2016-03-07T11:26:16.000+01:00</dateCommande> <horodatageCreation>2016-03-07T11:26:16.000+01:00</horodatageCreation> <horodatageModification>2016-03-07T14:09:24.803+01:00</horodatageModification> <contact>outiztest+wuwu@gmail.com</contact> <coupon>true</coupon> <codeCoupon>COUPON MAI</codeCoupon> <codePromotion>199</codePromotion> <nomCoupon>1000 utilisation</nomCoupon> <reductionCouponTTC>0.0</reductionCouponTTC> <reductionCouponHT>0.0</reductionCouponHT> <reductionCouponHTPercent>0.00</reductionCouponHTPercent> <montantBundle>0</montantBundle> <montantReducBundle></montantReducBundle> <montantReducBundleTTC></montantReducBundleTTC> <lieuPayment>WEB</lieuPayment> <moyenPayment>VISA</moyenPayment> <canalVente>OUTIZ</canalVente>

and I want to get the first line with date ..., and the commande number on this example.
I write my filtre for this type of message as follows :
grok { match => { "message" => "%{YEAR:YR}-%{MONTHNUM:MNTNUM}-%{MONTHDAY:MNTDAY}[ ]%{HOUR:HR}:?%{MINUTE:MIN}:?%{SECOND:SEC} %{LOGLEVEL:LOG}(\s*)\[\]\[%{NOTSPACE:CLASS}\]((.*)(\n))*(\s*)<order>(\n)(\s*)(<numCommande>)%{GREEDYDATA:NUMCMD}(</numCommande>)" }
add_field => { "haspath" => "yes" "endpoint" => "%{NUMCMD}" } break_on_matchl => true add_tag => "orderCRM" remove_tag => ["_grokparsefailure"] }

and input as follows :
input { file { type => "HYBRIS" path => "C:\Users\O6665391\ELK\logstash-2.2.2\multiline.log" codec => multiline { pattern => "^%{TIMESTAMP_ISO8601}" negate => true what => "previous" }}}

the problem is that on grokdebug, my filter works but with the .conf file, it doesn't work.
I have no error message, but it does not work

6

he apparently there's no one to help me .
thank you:sweat:

7

Questions asked here are typically answered by volunteers and you should set your expectations accordingly.

I don't have time to help further in this thread. Maybe someone else can help out.

8

okay, it 's okay .
I contiuner my research on my side .

thanks anyway