julsss
March 22, 2019, 8:06am
1
Good day everyone,
I am new to this technology and I am trying to filter an xml file with the following elements.
Now, when I checked kibana. The first event that I parsed always include the parent tag
Due to this when I checked my logstash service it always encounter an error
[2019-03-22T15:47:14,641][WARN ][logstash.filters.xml ] Error parsing xml with XmlSimple {:source=>"message", :value=>"\n <Event\n xmlns="http://logging.apache.org/log4j/2.0/events\ " thread="scheduling-1" level="DEBUG" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">\n <Instant epochSecond="1552441415" nanoOfSecond="434000000">\n You rest at an inn.\n ", :exception=>#<REXML::ParseException: No close tag for /Events
Can someone please enlighten why I am encountering this scenario.
Here is my config for the filebeat.
document_type : xml
Thank you in advance.
adrisr
March 22, 2019, 9:38am
2
Hi,
There is an error in your multiline.pattern, it is currently not matching any lines. You should get rid of the caret ^.
I tested using this settings:
multiline.pattern: '<Event>'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: '</Event>'
exclude_lines: 'Events>'
and this sample file:
<Events>
<Event>
<Message>Line 1</Message>
</Event>
<Event>
<Message>Line 2</Message>
</Event>
<Event>
<Message>Line 3</Message>
</Event>
</Events>
This results in 3 documents being published:
"message": " <Event>\n <Message>Line 1</Message>\n </Event>",
"message": " <Event>\n <Message>Line 2</Message>\n </Event>",
"message": " <Event>\n <Message>Line 3</Message>\n </Event>",
julsss
March 22, 2019, 10:09am
3
Helllo,
Thank you for the response. I tried the config you have provided. It removes the <Events> tag. But apparently the first <Event> tag is not included in the document published. This is the whole xml document I am using.
> <Events>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="DEBUG" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441415" nanoOfSecond="434000000"></Instant>
> <Message>You rest at an inn.</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You levelled up! You are now level 1</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You levelled up! You are now level 2</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You levelled up! You are now level 3</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You levelled up! You are now level 4</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You levelled up! You are now level 5</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You died</Message>
> </Event>
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="ERROR" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441417" nanoOfSecond="306000000"></Instant>
> <Message>You respawn</Message>
> </Event>
> </Events>
The document results is only 7.
The event below was not included in the document result:
> <Event
> xmlns="http://logging.apache.org/log4j/2.0/events" thread="scheduling-1" level="DEBUG" loggerName="com.ck.logdemo.AdventureTime" endOfBatch="false" loggerFqcn="org.apache.logging.log4j.spi.AbstractLogger" threadId="33" threadPriority="5">
> <Instant epochSecond="1552441415" nanoOfSecond="434000000"></Instant>
> <Message>You rest at an inn.</Message>
> </Event>
adrisr
March 22, 2019, 10:45am
4
Oh, I didn't realise that your "" is not found in a single line, so the provided multiline.pattern never matches.
Replace with:
multiline.pattern: '<Event'
julsss
March 25, 2019, 2:21am
5
Thanks for the response. That solved my problem.
May I know why the <Events> tag was included in the first document? Even though I set the multiline.match to after.
system
April 22, 2019, 2:21am
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
