Best filter to match xml element and flag value


(Major_Slackr) #1

Hi Guys,

Apologies if this is a bit of a noddy question (I am a noob to this) but I
couldn't find anything in the docs or the great G that really gave me a
good answer to this problem....

In some of my logs I have events containing xml. I want to be able to
identify the value of an element and create a new field containing the
value.

Example event in log:

INFO 2013-09-11 10:52:03,381 [[SmokeTest_0906-v20130906-10+25].x_to_bss_evt
.stage1.22] (TranslateDealId.java:50) - Received raw message from x: <?xml
version="1.0" encoding="utf-8"?><rm:Message SchemaVersion="1.09" xmlns:rm=
"xyz.com/rm/evt-title-updated">EVT-Title-Updated<
/MessageType>XY
115200591009311965115200591009311965</MessageID

.....etc

I want to be able to create a field CORRELATIONID from the element
115200591009311965 and have it populate the
value of the element into the field.

So, in this case the CORRELATIONID flag vould have a value 1
15200591009311965,

I had a look at grok methods using regex but could not see a way to do it.

Any suggestions or help much appreciated....

Cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Alexander Reelsen) #2

Hey,

as elasticsearch is only parsing JSON, you would have to convert the XML to
JSON before indexing it into elasticsearch. One possibility is to do this
in your application code, the other way would be using regexes as you
mentioned. See pattern replace char filter might be what you are searching
for. See
http://www.elasticsearch.org/guide/reference/index-modules/analysis/pattern-replace-charfilter/

--Alex

On Wed, Sep 11, 2013 at 1:53 PM, Major_Slackr
pthagonalthoughts@gmail.comwrote:

Hi Guys,

Apologies if this is a bit of a noddy question (I am a noob to this) but I
couldn't find anything in the docs or the great G that really gave me a
good answer to this problem....

In some of my logs I have events containing xml. I want to be able to
identify the value of an element and create a new field containing the
value.

Example event in log:

INFO 2013-09-11 10:52:03,381 [[SmokeTest_0906-v20130906-10+25].
x_to_bss_evt.stage1.22] (TranslateDealId.java:50) - Received raw message
from x: <?xml version="1.0" encoding="utf-8"?><rm:Message SchemaVersion=
"1.09" xmlns:rm="xyz.com/rm/evt-title-updated">EVT-
Title-UpdatedXY<
/Target>115200591009311965
115200591009311965.....etc

I want to be able to create a field CORRELATIONID from the element
115200591009311965 and have it populate
the value of the element into the field.

So, in this case the CORRELATIONID flag vould have a value 1
15200591009311965,

I had a look at grok methods using regex but could not see a way to do it.

Any suggestions or help much appreciated....

Cheers!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(David Pilato) #3

@Major_Slackr: is your question about logstash?

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr | @scrutmydocs

Le 12 sept. 2013 à 17:19, Alexander Reelsen alr@spinscale.de a écrit :

Hey,

as elasticsearch is only parsing JSON, you would have to convert the XML to JSON before indexing it into elasticsearch. One possibility is to do this in your application code, the other way would be using regexes as you mentioned. See pattern replace char filter might be what you are searching for. See http://www.elasticsearch.org/guide/reference/index-modules/analysis/pattern-replace-charfilter/

--Alex

On Wed, Sep 11, 2013 at 1:53 PM, Major_Slackr pthagonalthoughts@gmail.com wrote:
Hi Guys,

Apologies if this is a bit of a noddy question (I am a noob to this) but I couldn't find anything in the docs or the great G that really gave me a good answer to this problem....

In some of my logs I have events containing xml. I want to be able to identify the value of an element and create a new field containing the value.

Example event in log:

INFO 2013-09-11 10:52:03,381 [[SmokeTest_0906-v20130906-10+25].x_to_bss_evt.stage1.22] (TranslateDealId.java:50) - Received raw message from x: <?xml version="1.0" encoding="utf-8"?><rm:Message SchemaVersion="1.09" xmlns:rm="xyz.com/rm/evt-title-updated">EVT-Title-UpdatedXY115200591009311965115200591009311965.....etc

I want to be able to create a field CORRELATIONID from the element 115200591009311965 and have it populate the value of the element into the field.

So, in this case the CORRELATIONID flag vould have a value 115200591009311965,

I had a look at grok methods using regex but could not see a way to do it.

Any suggestions or help much appreciated....

Cheers!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #4