Logstash split xml filter

Elastic Stack Logstash
1

Hi all,
I have a some problem with my logstash filter.
I read data from an xml and I want to split them to send to elasticsearch. The xml is like this:

  <mounts>
       <mount>
           <name>/Subasio.aac</name>
           <up_time_seconds>59637</up_time_seconds>
           <active_clients count="59">
              <client>
                 <ip>79.53.52.154</ip>
                 <connection_time>1582093968</connection_time>
                 <request>GET /Subasio.aac HTTP/1.0</request>
                 <user_agent>Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.68 Safari/537.36 CrKey/1.44.191160</user_agent>
              </client>
              <client>
                 <ip>79.3.34.44</ip>
                 <connection_time>1582094940</connection_time>
                 <request>GET /Subasio.aac HTTP/1.0</request>
                 <user_agent>Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.68 Safari/537.36 CrKey/1.44.191160</user_agent>
             </client>
            ...
          </active_clients>
        </mount>
        <mount>
        ...
        </mount>

and so on.
My filter is as follow:

filter {
   xml {
    remove_namespaces => true
    source => "message"
    target => "radio"
   }

   split {
    field => "[radio][mounts]"
   }

   split {
    field => "[radio][mounts][mount]"
   }

   mutate {
     add_field => {
      stationName => "%{[radio][mounts][mount][name]}"
      ip => "%{[radio][mounts][mount][active_clients][count][client][ip]}"
     }

The result I expect is something like this

{
  "@timestamp" => 2020-02-19T15:53:01.793Z,
  "stationName" => "Subasio.aac",
  "ip" => "79.53.52.154",
  "@version" => "1"
}
{
  "@timestamp" => 2020-02-19T15:53:01.793Z,
  "stationName" => "Subasio.aac",
  "ip" => "79.3.34.44",
  "@version" => "1"
}

but I split only "stationName" and not ip, can someone help me please?
There are many mount name in the xml and every name has a lot of client with different ip.
Thank you

2

Try

split { field => "[radio][mount]" }
split { field => "[radio][mount][active_clients][0][client]" }

mutate {
    add_field => {
        stationName => "%{[radio][mount][name]}"
        ip => "%{[radio][mount][active_clients][0][client][ip][0]}"
    }
}
3

Thank you Badger but it not works. This doesn't split even the station name

{
           "tags" => [
        [0] "_split_type_failure"
    ],
       "@version" => "1",
             "ip" => "%{[radio][mount][active_clients][0][client][ip][0]}",
    "stationName" => "%{[radio][mount][name]}",
     "@timestamp" => 2020-02-20T08:55:01.749Z
}
[2020-02-20T09:56:14,256][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:[radio][mount] is of type = NilClass
[2020-02-20T09:56:14,262][WARN ][logstash.filters.split   ] Only String and Array types are splittable. field:[radio][mount][active_clients][0][client] is of type = NilClass
4

The XML you posted is not valid, so I had to make some assumptions about what it looked like. If the filter does not parse it then I infer than those assumptions were incorrect. If you can post valid, complete XML then I could adjust the assumptions.

5

Hi Badger the xml is correct. As you can see the mount name Subasio.aac has 59 active_clients with different ip. Another mount name has some active_clients with different ip.
There are more than 200 mounts and everyone has a lots of clients.
I wish to split all the 59 clients for Suabasio.aac mount name and so on.
Thank you for your time

6

When I take this XML

 <mounts>
       <mount>
           <name>/Subasio.aac</name>
           <up_time_seconds>59637</up_time_seconds>
           <active_clients count="59">
              <client>
                 <ip>79.53.52.154</ip>
                 <connection_time>1582093968</connection_time>
                 <request>GET /Subasio.aac HTTP/1.0</request>
                 <user_agent>Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.68 Safari/537.36 CrKey/1.44.191160</user_agent>
              </client>
              <client>
                 <ip>79.3.34.44</ip>
                 <connection_time>1582094940</connection_time>
                 <request>GET /Subasio.aac HTTP/1.0</request>
                 <user_agent>Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.68 Safari/537.36 CrKey/1.44.191160</user_agent>
             </client>
          </active_clients>
        </mount>
       <mount>
           <name>/Subasio.txt</name>
           <up_time_seconds>59637</up_time_seconds>
           <active_clients count="59">
              <client>
                 <ip>79.53.51.154</ip>
                 <connection_time>1582093968</connection_time>
                 <request>GET /Subasio.aac HTTP/1.0</request>
                 <user_agent>Mozilla/5.0 (X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.68 Safari/537.36 CrKey/1.44.191160</user_agent>
              </client>
          </active_clients>
        </mount>
</mounts>

and run it through

xml {
       remove_namespaces => true
       source => "message"
       target => "radio"
       remove_field => [ "message" ]
   }

split { field => "[radio][mount]" }
split { field => "[radio][mount][active_clients][0][client]" }

mutate {
    add_field => {
        stationName => "%{[radio][mount][name]}"
        ip => "%{[radio][mount][active_clients][0][client][ip][0]}"
    }
    remove_field => [ "radio" ]
}

I get

{
 "@timestamp" => 2020-02-20T17:59:24.780Z,
"stationName" => "/Subasio.aac",
         "ip" => "79.53.52.154"
}
{
 "@timestamp" => 2020-02-20T17:59:24.780Z,
"stationName" => "/Subasio.aac",
         "ip" => "79.3.34.44"
}
{
 "@timestamp" => 2020-02-20T17:59:24.780Z,
"stationName" => "/Subasio.txt",
         "ip" => "79.53.51.154"
}
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.