Parse multiline xml via syslog

Elastic Stack Logstash
21

Thanks Magnus! I am getting closer...

I am now unable to parse the field

<counter-name>

any ideas on what I am doing wrong here? It tried multiple variations using different xml paths.

Here is my new filter:

input {
  stdin {
    codec => multiline {
      pattern => "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL}"
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
  }
}
}

filter {

   grok {
	patterns_dir => "./patterns"
	match => ["message", "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL} \[Thread-4\] \(LogResponse:logResponse\) - %{GREEDYDATA:data}"]
}

   xml {
        store_xml => "false"
	remove_namespaces => "true"
        source => "data"
        xpath => [
            "//ssc/response/service/text()", "service",
	    "//generic-device-response/ptnii-equip-name/text()", "router",
	    "//filter-information/counter/counter-name/text()",  "address"
        ]
    }
}


output {
    stdout { codec => rubydebug }
}

and here is the output I receive, it does parse service (used this as simple test) and router name.

{
    "router" => [
    [0] "ROUTER"
],
"@timestamp" => 2018-06-24T19:00:15.229Z,
      "data" => "<ssc xmlns=\"http://123.com/security/ssc\">\n  <response>\n    <service>ddos</service>\n    <state>\n      <status>complete</status>\n    </state>\n    <query-response>\n      <generic-device-responses>\n        <generic-device-response>\n          <ptnii-equip-name>DTRMI411ME6</ptnii-equip-name>\n          <raw-device-responses>\n            <raw-device-response><![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name>__flowspec_default_inet__</filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>><![CDATA[]]></raw-device-response>\n          </raw-device-responses>\n        </generic-device-response>\n      </generic-device-responses>\n    </query-response>\n  </response>\n</ssc>",
   "service" => [
    [0] "ddos"
],
  "@version" => "1",
      "host" => "shdw01tool.ddostier4.att.net",
   "message" => "2018 Jun 14 16:56:32,305  INFO [Thread-4] (LogResponse:logResponse) - <ssc xmlns=\"http://123.com/security/ssc\">\n  <response>\n    <service>ddos</service>\n    <state>\n      <status>complete</status>\n    </state>\n    <query-response>\n      <generic-device-responses>\n        <generic-device-response>\n          <ptnii-equip-name>ROUTER</ptnii-equip-name>\n          <raw-device-responses>\n            <raw-device-response><![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name>__flowspec_default_inet__</filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>><![CDATA[]]></raw-device-response>\n          </raw-device-responses>\n        </generic-device-response>\n      </generic-device-responses>\n    </query-response>\n  </response>\n</ssc>",
      "tags" => [
    [0] "multiline"
]
}
22

The counter-name element is inside a nested XML document stored in a CDATA section in (AFAICT) /ssc/response/query-response/generic-device-responses/generic-device-response/raw-device-responses/raw-device-response. Extract that into a field and run the xml filter on it.

23

Hi Magnus,

I am sorry but could you please elaborate a little more on how I would extract CDATA into a field?

thanks!

24

I'm not sure how to describe it differently. Use your current xml filter to extract /ssc/response/query-response/generic-device-responses/generic-device-response/raw-device-responses/raw-device-response/text() (or something like that; my XPath skills are limited) to a field, then feed that field to another xml filter that extracts the counter-name field using some other XPath expression (/rpc-reply/firewall-information/filter-information/filter-name/counter/counter-name/text() or similar).

25

wow, ok, I understand now, thanks for explaining. Let me give it a whirl.

thanks!

26

OK, so now I trying the two xml filters but I am receiving an _xmlparsefailure. I am not sure if this will skew my results or not so I want to make sure I am using the proper logic in my logstash configuration. Please advise.

input {
  stdin {
    codec => multiline {
      pattern => "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL}"
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
  }
}
}

filter {

   grok {
        patterns_dir => "./patterns"
        match => ["message", "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL} \[Thread-4\] \(LogResponse:logResponse\) - %{GREEDYDATA:data}"]
}

   xml {
        store_xml => "false"
        remove_namespaces => "true"
        source => "data"
        xpath => [
            "//ssc/response/service/text()", "service",
            "//generic-device-response/ptnii-equip-name/text()", "router",
            "//ssc/response/query-response/generic-device-responses/generic-device-response/raw-device-responses/raw-device-response/text()", "data2"

        ]

    }

   xml {
        store_xml => "false"
        remove_namespaces => "true"
        source => "data2"
        xpath => [
           "//rpc-reply/firewall-information/filter-information/filter-name/counter/counter-name/text()", "address"
        ]
}


}


output {
    stdout { codec => rubydebug }
}

and now here is the output I receive. I had to truncate the output since this message would be too large...

],
     "data2" => [
    [0] "<![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name>__flowspec_default_inet__</filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>",
    [1] "&gt;"
],
  "@version" => "1",
      "host" => "shdw01tool.ddostier4.att.net",
   "message" => "2018 Jun 14 16:56:32,305  INFO [Thread-4] (LogResponse:logResponse) - <ssc xmlns=\"http://123.com/security/ssc\">\n  <response>\n    <service>ddos</service>\n    <state>\n      <status>complete</status>\n    </state>\n    <query-response>\n      <generic-device-responses>\n        <generic-device-response>\n          <ptnii-equip-name>ROUTER</ptnii-equip-name>\n          <raw-device-responses>\n            <raw-device-response><![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name>__flowspec_default_inet__</filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>><![CDATA[]]></raw-device-response>\n          </raw-device-responses>\n        </generic-device-response>\n      </generic-device-responses>\n    </query-response>\n  </response>\n</ssc>",
      "tags" => [
    [0] "multiline",
    [1] "_xmlparsefailure"
]

}

27

Okay, it looks like you're on the right track. Two obvious problems that I don't know OTOH how to solve:

  • The <!CDATA stuff. Is there an XPath way of peeling away that stuff?
  • What's up with the &gt; in the second element of the data2 field? It's not clear to me what it comes from. Worst case you can use a mutate filter to replace data2 with [data2][0], i.e. effectively turn data2 into a string field based on the first element of today's array.
28

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.