# Parse multiline xml via syslog

**URL:** <https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138>\
**Category:** Logstash\
**Created:** [June 15, 2018, 8:43pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138 "2018-06-15T20:43:12Z")\
**Posts on this page:** 8\
**Page:** 2

<div class="post-metadata">

**Author:** ![Eric\_Mueller](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@Eric\_Mueller](https://discuss.elastic.co/u/Eric_Mueller)\
**Post date:** [June 24, 2018, 7:05pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/21 "2018-06-24T19:05:53Z")

</div>

Thanks Magnus! I am getting closer...

I am now unable to parse the field

`<counter-name>`

any ideas on what I am doing wrong here? It tried multiple variations using different xml paths.

Here is my new filter:

```
input {
  stdin {
    codec => multiline {
      pattern => "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL}"
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
  }
}
}

filter {

   grok {
	patterns_dir => "./patterns"
	match => ["message", "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL} \[Thread-4\] \(LogResponse:logResponse\) - %{GREEDYDATA:data}"]
}

   xml {
        store_xml => "false"
	remove_namespaces => "true"
        source => "data"
        xpath => [
            "//ssc/response/service/text()", "service",
	    "//generic-device-response/ptnii-equip-name/text()", "router",
	    "//filter-information/counter/counter-name/text()", "address"
        ]
    }
}

output {
    stdout { codec => rubydebug }
}

```

and here is the output I receive, it does parse service (used this as simple test) and router name.

```
{
    "router" => [
    [0] "ROUTER"
],
"@timestamp" => 2018-06-24T19:00:15.229Z,
      "data" => "<ssc xmlns=\"http://123.com/security/ssc\">\n <response>\n <service>ddos</service>\n <state>\n <status>complete</status>\n </state>\n <query-response>\n <generic-device-responses>\n <generic-device-response>\n <ptnii-equip-name>DTRMI411ME6</ptnii-equip-name>\n <raw-device-responses>\n <raw-device-response><![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name> __flowspec_default_inet__ </filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>><![CDATA[]]></raw-device-response>\n </raw-device-responses>\n </generic-device-response>\n </generic-device-responses>\n </query-response>\n </response>\n</ssc>",
   "service" => [
    [0] "ddos"
],
  "@version" => "1",
      "host" => "shdw01tool.ddostier4.att.net",
   "message" => "2018 Jun 14 16:56:32,305 INFO [Thread-4] (LogResponse:logResponse) - <ssc xmlns=\"http://123.com/security/ssc\">\n <response>\n <service>ddos</service>\n <state>\n <status>complete</status>\n </state>\n <query-response>\n <generic-device-responses>\n <generic-device-response>\n <ptnii-equip-name>ROUTER</ptnii-equip-name>\n <raw-device-responses>\n <raw-device-response><![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name> __flowspec_default_inet__ </filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>><![CDATA[]]></raw-device-response>\n </raw-device-responses>\n </generic-device-response>\n </generic-device-responses>\n </query-response>\n </response>\n</ssc>",
      "tags" => [
    [0] "multiline"
]
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 24, 2018, 8:57pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/22 "2018-06-24T20:57:40Z")

</div>

The counter-name element is inside a nested XML document stored in a CDATA section in (AFAICT) /ssc/response/query-response/generic-device-responses/generic-device-response/raw-device-responses/raw-device-response. Extract that into a field and run the xml filter on it.

---

<div class="post-metadata">

**Author:** ![Eric\_Mueller](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@Eric\_Mueller](https://discuss.elastic.co/u/Eric_Mueller)\
**Post date:** [June 25, 2018, 12:53pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/23 "2018-06-25T12:53:24Z")

</div>

Hi Magnus,

I am sorry but could you please elaborate a little more on how I would extract CDATA into a field?

thanks!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 25, 2018, 1:08pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/24 "2018-06-25T13:08:25Z")

</div>

I'm not sure how to describe it differently. Use your current xml filter to extract /ssc/response/query-response/generic-device-responses/generic-device-response/raw-device-responses/raw-device-response/text() (or something like that; my XPath skills are limited) to a field, then feed that field to another xml filter that extracts the counter-name field using some other XPath expression (/rpc-reply/firewall-information/filter-information/filter-name/counter/counter-name/text() or similar).

---

<div class="post-metadata">

**Author:** ![Eric\_Mueller](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@Eric\_Mueller](https://discuss.elastic.co/u/Eric_Mueller)\
**Post date:** [June 25, 2018, 1:22pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/25 "2018-06-25T13:22:27Z")

</div>

wow, ok, I understand now, thanks for explaining. Let me give it a whirl.

thanks!

---

<div class="post-metadata">

**Author:** ![Eric\_Mueller](https://avatars.discourse-cdn.com/v4/letter/e/6f9a4e/32.png) [@Eric\_Mueller](https://discuss.elastic.co/u/Eric_Mueller)\
**Post date:** [June 26, 2018, 1:18pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/26 "2018-06-26T13:18:03Z")

</div>

OK, so now I trying the two xml filters but I am receiving an \_xmlparsefailure. I am not sure if this will skew my results or not so I want to make sure I am using the proper logic in my logstash configuration. Please advise.

```
input {
  stdin {
    codec => multiline {
      pattern => "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL}"
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
  }
}
}

filter {

   grok {
        patterns_dir => "./patterns"
        match => ["message", "^%{YEAR} %{MONTH}\s+%{MONTHDAY} %{TIME}\s+%{LOGLEVEL} \[Thread-4\] \(LogResponse:logResponse\) - %{GREEDYDATA:data}"]
}

   xml {
        store_xml => "false"
        remove_namespaces => "true"
        source => "data"
        xpath => [
            "//ssc/response/service/text()", "service",
            "//generic-device-response/ptnii-equip-name/text()", "router",
            "//ssc/response/query-response/generic-device-responses/generic-device-response/raw-device-responses/raw-device-response/text()", "data2"

        ]

    }

   xml {
        store_xml => "false"
        remove_namespaces => "true"
        source => "data2"
        xpath => [
           "//rpc-reply/firewall-information/filter-information/filter-name/counter/counter-name/text()", "address"
        ]
}

}

output {
    stdout { codec => rubydebug }
}

```

and now here is the output I receive. I had to truncate the output since this message would be too large...

> ```
> ],
> "data2" => [
> [0] "<![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name> __flowspec_default_inet__ </filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>",
> [1] "&gt;"
> ],
> "@version" => "1",
> "host" => "shdw01tool.ddostier4.att.net",
> "message" => "2018 Jun 14 16:56:32,305 INFO [Thread-4] (LogResponse:logResponse) - <ssc xmlns=\"http://123.com/security/ssc\">\n <response>\n <service>ddos</service>\n <state>\n <status>complete</status>\n </state>\n <query-response>\n <generic-device-responses>\n <generic-device-response>\n <ptnii-equip-name>ROUTER</ptnii-equip-name>\n <raw-device-responses>\n <raw-device-response><![CDATA[<![CDATA[<?xml version=\"1.0\" encoding=\"UTF-8\"?><rpc-reply xmlns:junos=\"http://xml.juniper.net/junos/15.1F5/junos\">\n<firewall-information xmlns=\"http://xml.juniper.net/junos/15.1F5/junos-filter\">\n<filter-information>\n<filter-name> __flowspec_default_inet__ </filter-name>\n<counter>\n<counter-name>1.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.0.12.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.25,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>1.125.161.5,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.0.0.0,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.10.11.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.100.100.49,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.105.6.200,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.11.1.1,*</counter-name>\n<packet-count>14691295769</packet-count>\n<byte-count>21713735146582</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.0.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n<counter>\n<counter-name>10.15.1.1,*</counter-name>\n<packet-count>0</packet-count>\n<byte-count>0</byte-count>\n</counter>\n</filter-information>\n</firewall-information>\n</rpc-reply>]]]]>><![CDATA[]]></raw-device-response>\n </raw-device-responses>\n </generic-device-response>\n </generic-device-responses>\n </query-response>\n </response>\n</ssc>",
> "tags" => [
> [0] "multiline",
> [1] "_xmlparsefailure"
> ]
> 
> ```
> 
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 26, 2018, 1:40pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/27 "2018-06-26T13:40:18Z")

</div>

Okay, it looks like you're on the right track. Two obvious problems that I don't know OTOH how to solve:

- The \<!CDATA stuff. Is there an XPath way of peeling away that stuff?
- What's up with the `&gt;` in the second element of the `data2` field? It's not clear to me what it comes from. Worst case you can use a mutate filter to replace `data2` with `[data2][0]`, i.e. effectively turn `data2` into a string field based on the first element of today's array.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 24, 2018, 1:40pm UTC](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138/28 "2018-07-24T13:40:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/parse-multiline-xml-via-syslog/136138.md?page=1)