Logstash xml plugin not working

[muireann (Muireann Mac Carthy)]

hello,

I'm trying to use the below logstash config to read from an xml file. The xml is being read into logstash as I can see the different lines from the xml printed in the "message" in the console when I run logstash.

input {
  file{
  path => "C:/xmlFiles/testFile.xml"
  start_position => "beginning"
  sincedb_path => "NUL"
    }  
}
filter {
	xml {
		source => "message"
		force_array => "false"
		add_tag => "Test Data"
                xpath => ["/report/counter[@type='CLASS']/@tested", "already-tested"]
		xpath => ["/report/counter[@type='CLASS']/@totest", "to-test"]
		store_xml => false
		remove_field => ["message"]       
	}
	  mutate{	
		replace => { "to-test" => "%{to-test[0]}" }
		replace => { "already-tested" => "%{already-tested[0]}" }
		}	 
}
  
output {
   elasticsearch {
      hosts => ["localhost:9200"]
      index => "testingxmlindex"
   }
     stdout { codec => rubydebug }
}

The mutate / xml plugin appears not to be working however, as the values that are added to my local kibana and printed in the console are "to-test[0]" instead of the actual number (262).

I have tested the xml and xpath I am using on the following site and both appear to work fine on here: https://www.freeformatter.com/xpath-tester.html

anyone any idea why this might be happening in logstash?

XML File:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<report>
    <counter type="TOTALCOUNT" totest="3563" tested="6436"/>
    <counter type="MISSING" totest="3636" tested="3634"/>
    <counter type="LINES" totest="734" tested="114"/>
    <counter type="COMPLETED" totest="8448" tested="4362"/>
    <counter type="TOCOUNT" totest="258" tested="5737"/>
    <counter type="CLASS" totest="262" tested="8653"/>
</report>

[Badger]

If the XML is spread across multiple lines you would need to use a multiline codec to combine all the lines in one XML document into a single event. See this for an example.

[muireann (Muireann Mac Carthy)]

Hi Badger,

Thanks for the response.

using a mulitline codec didn't work for me however and I still got the same issue where the values are printed as {to-test[0]} instead of the actual value.

When I use the http input instead of file and send the exact same xml to logstash through postman it works however, wondering if you could explain why? as for my use case I need it to read directly from an xml file.

Config changed to:

input {
  http {
    host => "0.0.0.0"
    port => "8080"
  } 
}
filter {
	xml {
		source => "message"
		force_array => "false"
		add_tag => "Test Data"
            xpath => ["/report/counter[@type='CLASS']/@tested", "already-tested"]
		xpath => ["/report/counter[@type='CLASS']/@totest", "to-test"]
		store_xml => false
		remove_field => ["message"]       
	}
	  mutate{	
		replace => { "to-test" => "%{to-test[0]}" }
		replace => { "already-tested" => "%{already-tested[0]}" }
		}	 
}
  
output {
   elasticsearch {
  hosts => ["localhost:9200"]
  index => "testingxmlindex"
   }
 stdout { codec => rubydebug }
}

[Badger]

@muireann

"%{to-test[0]}"

I am surprised you do not get an exception for that. Try %{[to-test][0]}