Filtering in Logstash for XML data

Elastic Stack Logstash
1

Need help in writing the right conf file for the data displayed here. Getting the splittable errors

Input file --- from the url

<clusterMetrics>

<appsSubmitted>12434</appsSubmitted>

<appsCompleted>12279</appsCompleted>

<appsPending>0</appsPending>

<appsRunning>9</appsRunning>

<appsFailed>13</appsFailed>

<appsKilled>133</appsKilled>

<reservedMB>0</reservedMB>

<availableMB>1050624</availableMB>

<allocatedMB>147456</allocatedMB>

<reservedVirtualCores>0</reservedVirtualCores>

<availableVirtualCores>60</availableVirtualCores>

<allocatedVirtualCores>24</allocatedVirtualCores>

<containersAllocated>24</containersAllocated>

<containersReserved>0</containersReserved>

<containersPending>0</containersPending>

<totalMB>1198080</totalMB>

<totalVirtualCores>84</totalVirtualCores>

<totalNodes>3</totalNodes>

<lostNodes>0</lostNodes>

<unhealthyNodes>0</unhealthyNodes>

<decommissionedNodes>0</decommissionedNodes>

<rebootedNodes>0</rebootedNodes>

<activeNodes>3</activeNodes>

</clusterMetrics>

Code:

Input used

exec {

type =&gt; "metrics"

command =&gt; "curl -s http://13.11.xxx.234:8088/ws/v1/cluster/metrics"

interval =&gt; "5"

codec =&gt; "json"

}

filter used

if [type] == "metrics"

{

 split {

         field =&gt; "metrics[clusterMetrics]"

         remove_field =&gt; [ "command" ]

 }

}

Output used

elasticsearch {

index =&gt; "yarn-metrics-%{+YYYY.MM.dd}"

document_type =&gt; "doc"

codec =&gt; "json"

servername...

username

password..
}

2
  • Why run curl instead of using the http_poller input plugin?
  • If curl indeed returns XML you should remove codec => json.
  • To parse the XML use an xml filter.
  • The point of the split filter is unclear. That filter requires an array as input and transforming the XML in your example won't produce any arrays.
3

Hi Magnus,

Based on your suggestions I changed to http polar and also used the xml.

The code with Jason and also Xml, http polar all are working in 5.6 version of elastic but failing with 5.2.1 version.

It is throwing splittable errors

Regards,
Bharath

4

How to check the split filter value , can u suggest? I am writing the logstash conf for the first time

5

Instead of describing what you see, show us. What does an example event look like? Use a stdout { codec => rubydebug } output to dump the raw event.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.