Parse XML response and extract the attributes from it to form new fields

Elastic Stack Logstash
1

Hi There,

I am working on monitoring a specific queue in ActiveMQ server. For the same i am using the xml utility which returns the structure of the queue like below;

This XML file does not appear to have any style information associated with it. The document tree is shown below.

<queues>
  <queue name="Sample">
    <stats size="0" consumerCount="0" enqueueCount="1" dequeueCount="1"/>
    <feed>
      <atom>queueBrowse/Sample?view=rss&amp;feedType=atom_1.0</atom>
      <rss>queueBrowse/Sample?view=rss&amp;feedType=rss_2.0</rss>
    </feed>
  </queue>
  <queue name="PUBLISHEVENTS.UAT">
    <stats size="1" consumerCount="0" enqueueCount="2" dequeueCount="1"/>
    <feed>
      <atom>
queueBrowse/PUBLISHEVENTS.UAT?view=rss&amp;feedType=atom_1.0
</atom>
      <rss>
queueBrowse/PUBLISHEVENTS.UAT?view=rss&amp;feedType=rss_2.0
</rss>
    </feed>
  </queue>
  <queue name="Test">
    <stats size="1" consumerCount="0" enqueueCount="1" dequeueCount="0"/>
    <feed>
      <atom>queueBrowse/Test?view=rss&amp;feedType=atom_1.0</atom>
      <rss>queueBrowse/Test?view=rss&amp;feedType=rss_2.0</rss>
    </feed>
  </queue>
</queues>

Below is the conf file i am trying but unable to extract the attributes as expected;

input
{
	http_poller {
		urls => {
			test => {
				method => get
				url => "http://localhost:8161/admin/xml/queues.jsp"
				user => "admin"
				password => "admin"
				headers => {
					Accept => "application/json"
				}
			}
		}
		request_timeout => 60
		codec => "plain"
		schedule => { "every" => "30s" }
	}
}

filter {
	xml {
		source => "message"
		target => "parsed"
	}
	split {
		field => "[parsed][queue]"
		add_field => {
		queue_name     => "%{[parsed][queue][@name]}"
		queue_size     => "%{[parsed][queue][stats][@size]}"
		consumer_count => "%{[parsed][queue][stats][@consumerCount]}"
		enqueue_count  => "%{[parsed][queue][stats][@enqueueCount]}"
		dequeue_count  => "%{[parsed][queue][stats][@dequeueCount]}"
		}
		}
		
	mutate {
		#convert => {
		#  "queue_size"     => "integer"
		#  "consumer_count" => "integer"
		#  "enqueue_count"  => "integer"
		#  "dequeue_count"  => "integer"
		#}
		remove_field => [ "message", "host" ]
	}
	
}

output {
  stdout {
    codec => rubydebug
  }
}

Below is the console output;

{
      "@version" => "1",
 "enqueue_count" => "%{[parsed][queue][stats][@enqueueCount}]",
 "dequeue_count" => "%{[parsed][queue][stats][@dequeueCount}]",
"consumer_count" => "%{[parsed][queue][stats][@consumerCount]}",
    "@timestamp" => 2019-04-16T12:42:23.382Z,
    "queue_size" => "%{[parsed][queue][stats][@size]}",
    "queue_name" => "%{[parsed][queue][@name]}"
}
{
      "@version" => "1",
 "enqueue_count" => "%{[parsed][queue][stats][@enqueueCount}]",
 "dequeue_count" => "%{[parsed][queue][stats][@dequeueCount}]",
"consumer_count" => "%{[parsed][queue][stats][@consumerCount]}",
    "@timestamp" => 2019-04-16T12:42:23.382Z,
    "queue_size" => "%{[parsed][queue][stats][@size]}",
    "queue_name" => "%{[parsed][queue][@name]}"
}
{
      "@version" => "1",
 "enqueue_count" => "%{[parsed][queue][stats][@enqueueCount}]",
 "dequeue_count" => "%{[parsed][queue][stats][@dequeueCount}]",
"consumer_count" => "%{[parsed][queue][stats][@consumerCount]}",
    "@timestamp" => 2019-04-16T12:42:23.382Z,
    "queue_size" => "%{[parsed][queue][stats][@size]}",
    "queue_name" => "%{[parsed][queue][@name]}"
}

Please kindly help how we can extract the attributes!

Regards

Kaushik

2

Close, but not quite right. Remove the trailing ]. Remove the @. And stats is an array, so you need to provide an index.

consumer_count => "%{[parsed][queue][stats][0][consumerCount}"

Same changes for the others, except queue_name, which does not need an array index.

3

@Badger, thank for your time in replying to this. If i have to only get the stats of a specific queue, like say in the above XML i want to extract the data of only the queue name "PUBLISHEVENTS.UAT" any idea how do i go about it?
Note: The index may not be same all the time.

4

You could extract a specific element from the array using ruby

    ruby {
        code => '
            a = event.get("[@metadata][theXML][queue]")
            a.each_index { |x|
                if a[x]["name"] == "PUBLISHEVENTS.UAT"
                    event.set("queueStats", a[x])
                end
            }
        '
    }
5

Hi @Badger, I am yet to try your provided solution. However, i tried to achieve my requirement using drop plugin with an if condition as below;

if "PUBLISHEVENTS.UAT" not in [queue_name]
{
	drop {}
}

But i now need to add a files with the size of the

queues data like in the above xml data it is 3

Could you drop some insight over?

I actually have a python script which outputs a json by parsing the xml received in the above poller. Any insights?

Regards

Kaushik.

6

Insert this between the xml and split filters

ruby { code => 'event.set("numOfQueues", event.get("[parsed][queue]").length)' }
7

Works like a charm. :wink: I have checked the ruby filter plugin documentation but did not find it much helpful. Do you have any good references for ruby scripting?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.