Extract data with Logstash and Xpath

Elastic Stack Logstash
1

Hi ,
I want to extract data (timestamp and message) via Xpath plugin in Logstash from XML files to display only them in fields in kibana.

here is my LOGSTASH conf :

input {
beats {
port => 5044
}
}
filter{
xml{
target => "doc"
store_xml => false
source => "message"
xpath =>
["/E2ETraceEvent/System/EventID/@EventID", "event_id",
"/E2ETraceEvent/System/Type/@Type", "type",
"/E2ETraceEvent/System/SubType/@SubType", "name",
"/E2ETraceEvent/System/Level/@Level", "level",
"/E2ETraceEvent/System/TimeCreated/@TimeCreated", "time"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "100.101.15.181:9200"
manage_template => false
index => "t11-%{+YYYY.MM}"
}
}
in my conf I'm tryig to parse only couple of columns because of the complexity of log, I need to parse all the log data.

I'm on it couple of day and i don't get any errors on logstash log or filebeat but no data inserted into ELASTIC.

can someone help me to understand how to work it out?

Thanks in advance.

2

That is not XML. Can you post the XML between line contains three backticks, like this: ```

2 Likes
3

To pull out the value of individual nodes, use

xpath => { "/E2ETraceEvent/System/EventID/text()" => "event_id" }

If you do that, you may also want to

if [event_id] { mutate { replace => { "event_id" => "%{[event_id][0]}" } } }

However, if you want to parse all of the xml, use

store_xml => true target => "someField"
3 Likes
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.