# How to parse xml log inside of the json format in Logstash

**URL:** <https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006>\
**Category:** Logstash\
**Created:** [November 24, 2021, 3:49am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006 "2021-11-24T03:49:48Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [November 24, 2021, 3:49am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/1 "2021-11-24T03:49:48Z")

</div>

I will receive a log that contained JSON and XML format  
I configurated the Logstash to receive TCP in JSON and used the filter when shown dateparsefailure in tags.  
The JSON part is able to extract, but the xml part can't.  
Could anyone give me some advice?  
Thanks

 ![Screenshot 2021-11-24 at 11.48.16 AM](https://us1.discourse-cdn.com/elastic/original/3X/c/a/cab82e20d6017f7f88b367d834dc6cf5fde19c7b.png)  
 ![Screenshot 2021-11-24 at 11.50.12 AM](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a5b2df7b224c2adac1b09300bc9eb865fd74ca06.png)

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [November 29, 2021, 3:24am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/2 "2021-11-29T03:24:06Z")

</div>

Anyone have idea?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 29, 2021, 3:53am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/3 "2021-11-29T03:53:39Z")

</div>

Please do not post pictures of text. They cannot be searched, are inaccessible to some people and I cannot copy and paste them to experiment with how to fix any problems. Just post the text.

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [November 29, 2021, 5:35am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/4 "2021-11-29T05:35:44Z")

</div>

Sorry, my fault.  
This is the Log in message

`<Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task>5</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-11-24T10:00:00.624971000Z"/><EventRecordID>227402</EventRecordID><Correlation/><Execution ProcessID="1095" ThreadID="1095"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ubuntu</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-11-23 02:56:47.183</Data><Data Name="ProcessGuid">{3b95acb1-4b99-619b-9535-d747b4550000}</Data><Data Name="ProcessId">732</Data><Data Name="Image">/usr/sbin/multipathd</Data><Data Name="User">root</Data></EventData></Event>`

here is the filter.conf

```auto
filter {
   if "_dateparsefailure" in [tags] {
     xml {
        store_xml => "false"
        source => "Message"

        xpath => [
          "//Event/System/EventID/text()","Event_id"
        ]
     }
   }
}

```

logstash input conf

```auto
input {
  tcp {
    codec => json_lines { charset => CP1252 }
    port => "5444"
  }
}

filter {
  date {
    locale => "en"
    timezone => "Etc/GMT"
    match => ["EventTime", "YYYY-MM-dd HH:mm:ss"]
  }
}

output {
   elasticsearch {
    hosts => "http://localhost:9200"
    index => "nxlog-linux-%{+YYYY.MM.dd}"
   }
   stdout { codec => rubydebug }
}

```

Thanks for the help.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 29, 2021, 5:52pm UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/5 "2021-11-29T17:52:28Z")

</div>

If I run logstash with

```
input { generator { count => 1 lines => ['<Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task>5</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-11-24T10:00:00.624971000Z"/><EventRecordID>227402</EventRecordID><Correlation/><Execution ProcessID="1095" ThreadID="1095"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ubuntu</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-11-23 02:56:47.183</Data><Data Name="ProcessGuid">{3b95acb1-4b99-619b-9535-d747b4550000}</Data><Data Name="ProcessId">732</Data><Data Name="Image">/usr/sbin/multipathd</Data><Data Name="User">root</Data></EventData></Event>'] } }
filter {
    xml {
        store_xml => "false"
        source => "message"
        xpath => { "//Event/System/EventID/text()" => "Event_id" }
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

```

then I get

```
  "Event_id" => [
    [0] "5"
],

```

It looks OK to me.

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [November 30, 2021, 5:32am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/6 "2021-11-30T05:32:39Z")

</div>

Thanks for the reply.  
As I am forwarding the JSON log to Logstash using tcp, the whole message is shown as below:

```auto
Nov 30 05:28:02 ubuntu sysmon: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>1</EventID><Version>5</Version><Level>4</Level><Task>1</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-11-30T05:28:02.358007000Z"/><EventRecordID>274376</EventRecordID><Correlation/><Execution ProcessID="1095" ThreadID="1095"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ubuntu</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-11-24 07:14:26.940</Data><Data Name="ProcessGuid">{3b95acb1-e652-619d-31b4-afeedb550000}</Data><Data Name="ProcessId">77439</Data><Data Name="Image">/usr/bin/cat</Data><Data Name="FileVersion">-</Data><Data Name="Description">-</Data><Data Name="Product">-</Data><Data Name="Company">-</Data><Data Name="OriginalFileName">-</Data><Data Name="CommandLine">cat syslog</Data><Data Name="CurrentDirectory">/home/ubuntu</Data><Data Name="User">ubuntu</Data><Data Name="LogonGuid">{3b95acb1-b63f-61a5-e803-000000000000}</Data><Data Name="LogonId">1000</Data><Data Name="TerminalSessionId">68</Data><Data Name="IntegrityLevel">no level</Data><Data Name="Hashes">-</Data><Data Name="ParentProcessGuid">{3b95acb1-e62f-619d-05d7-79d127560000}</Data><Data Name="ParentProcessId">77404</Data><Data Name="ParentImage">/usr/bin/bash</Data><Data Name="ParentCommandLine">-bash</Data><Data Name="ParentUser">ubuntu</Data></EventData></Event>
Nov 30 05:28:02 ubuntu sysmon: <Event><System><Provider Name="Linux-Sysmon" Guid="{ff032593-a8d3-4f13-b0d6-01fc615a0f97}"/><EventID>5</EventID><Version>3</Version><Level>4</Level><Task>5</Task><Opcode>0</Opcode><Keywords>0x8000000000000000</Keywords><TimeCreated SystemTime="2021-11-30T05:28:02.359124000Z"/><EventRecordID>274377</EventRecordID><Correlation/><Execution ProcessID="1095" ThreadID="1095"/><Channel>Linux-Sysmon/Operational</Channel><Computer>ubuntu</Computer><Security UserId="0"/></System><EventData><Data Name="RuleName">-</Data><Data Name="UtcTime">2021-11-24 07:14:26.941</Data><Data Name="ProcessGuid">{3b95acb1-e652-619d-31b4-afeedb550000}</Data><Data Name="ProcessId">77439</Data><Data Name="Image">/usr/bin/cat</Data><Data Name="User">ubuntu</Data></EventData></Event>

```

For the message field that's XML and dynamic data, how can I set up the logstash.conf to extract the message field for dynamic?  
Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 30, 2021, 4:52pm UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/7 "2021-11-30T16:52:11Z")

</div>

You would use grok to extract the xml from the log line. But from your original kibana screenshot it looks like you have already done that.

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [December 1, 2021, 1:50am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/8 "2021-12-01T01:50:14Z")

</div>

As I am not familiar with using the grok, could you help to take an example of my case that how to extract the message field.  
Thanks a lot.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 1, 2021, 2:05am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/9 "2021-12-01T02:05:35Z")

</div>

You could try

```
grok { match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:hostname} %{WORD}: %{GREEDYDATA:theXML}" } }
```

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [December 1, 2021, 2:41am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/10 "2021-12-01T02:41:40Z")

</div>

```auto
filter {
  grok
   {
        match => { "message" => "%{SYSLOGTIMESTAMP:timestamp} %{WORD:hostname} %{WORD}: %{GREEDYDATA:Message}" }
   }
    xml {
        store_xml => "false"
        source => "Message"
        }
}

```

it show \_grokparsefailure, \_dateparsefailure.  
Should i receive the log via Json? or just let it as raw or xml?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 1, 2021, 3:29am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/11 "2021-12-01T03:29:10Z")

</div>

What does the [message] field look like? (Not the [Message] field, that is different.) How does the [Message] field get created?

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [December 1, 2021, 5:10am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/12 "2021-12-01T05:10:36Z")

</div>

The whole log looks like this:

 ![Screenshot 2021-12-01 at 1.05.32 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/2/322a29d3f2a8441efd7e74925da50a92a577055a.png)  
As the raw log is JSON format, so it could extract some parts of data. but there are some raw log which is in XML format that shows in the [Message] field, it should be created by parsing error

So, i just want to extract these XML log field

 ![Screenshot 2021-12-01 at 1.05.41 PM](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3ef4e77ff1557b089f525b4bd37a61c3fde288f1.png)

Thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 1, 2021, 5:20am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/13 "2021-12-01T05:20:28Z")

</div>

OK, so forget grok, just do

```
xml {
    store_xml => "false"
    source => "Message"
    xpath => { "//Event/System/EventID/text()" => "Event_id" }
}

```

to parse the [Message] (not [message]) field.

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [December 1, 2021, 5:39am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/14 "2021-12-01T05:39:36Z")

</div>

It works!  
Thanks bro.

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [December 1, 2021, 6:24am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/15 "2021-12-01T06:24:02Z")

</div>

Just one more things, How can i extract the data in

```auto
<Event>
   <EventData>
       <Data Name="LogonId">1000</Data>
   </EventData>
</Event>

```

Thanks a lots

---

<div class="post-metadata">

**Author:** ![Bigboy0706](https://avatars.discourse-cdn.com/v4/letter/b/82dd89/32.png) [@Bigboy0706](https://discuss.elastic.co/u/Bigboy0706)\
**Post date:** [December 1, 2021, 6:28am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/16 "2021-12-01T06:28:09Z")

</div>

Sorry, i just figure out the way.  
Thanks man.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 29, 2021, 6:28am UTC](https://discuss.elastic.co/t/how-to-parse-xml-log-inside-of-the-json-format-in-logstash/290006/17 "2021-12-29T06:28:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.