# Convert embede xml in log to json in logstash

**URL:** <https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094>\
**Category:** Logstash\
**Created:** [July 14, 2024, 10:03am UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094 "2024-07-14T10:03:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\
**Post date:** [July 14, 2024, 10:03am UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094/1 "2024-07-14T10:03:50Z")

</div>

logs sent from filebeat to logstash and I write a pattern to parse them.developers added logs that have embeded xml inside logs and here is a sample :

```auto
2024-07-08 15:18:35,608 INFO |body=<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<ss2:normal xmlns:ns2="http://service..com/">
    <context>
        <data>
            <key>ADDRESS</key>
            <value>0.0.0.0</value>
        </data>
        <data>
            <key>ssid</key>
            <value>d1d71e02-25ff-1</value>
        </data>
        <data>
            <key>m-id</key>
            <value>25440fb</value>
        </data>
    </context>
    <request>
        <amount>50</amount>
    </request>
</ss2:normal>

```

how can I parse body section and retrieve values in json format considering that xml in body is dynamiclaly changed by type of services?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 14, 2024, 11:52am UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094/2 "2024-07-14T11:52:23Z")

</div>

You could try something like

```
    if "|body=<?xml" in [message] {
        mutate { copy => { "message" => "[@metadata][message]" } }
        mutate { gsub => ["[@metadata][message]", "^.*\|body=", "" ] }
        # Remove namespaces like ss2:
        mutate { gsub => ["[@metadata][message]", "<(/?)\w+:", "<\1" ] }
        xml {
            source => "[@metadata][message]"
            target => "theXML"
            store_xml => true
            force_array => false
        }
        ruby {
            code => '
                data = event.remove("[theXML][context][data]")
                if data.is_a? Array
                    data.each { |x|
                        key = x["key"]
                        event.set("[theXML][context][#{key}]", x["value"])
                    }
                end
            '
        }
    }

```

Note that using ruby to convert the [context][data] array to hash values may result in an explosion of field names that causes downstream problems.

---

<div class="post-metadata">

**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\
**Post date:** [July 14, 2024, 12:16pm UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094/3 "2024-07-14T12:16:06Z")

</div>

I need log level and timestamp too , actually treat whole as a sinle log  
do I need to change or add any configuration in filebeat? for example multiline codec if so , how can I config filebeat for thi specific logs?  
currently my filebeat config is :

> - type: log  
> fields:  
> source: 'filebeat2'  
> logID: logbackup  
> fields\_under\_root: true  
> enabled: true  
> paths:
> - /home/logbackup/a.log
> - /home/logbackup/backup/a.log-\*  
> #- c:\programdata\elasticsearch\logs\*  
> #ignore\_older: 24h  
> close\_inactive : 30m

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 14, 2024, 12:29pm UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094/4 "2024-07-14T12:29:30Z")

</div>

> [@alex\_petrov](#):
>
> I need log level and timestamp too , actually treat whole as a sinle log  
> do I need to change or add any configuration in filebeat?

Yes, you do. I was using

```
file { 
    path => "/home/user/test.txt" 
    sincedb_path => "/dev/null" 
    start_position => beginning 
    codec => multiline { 
        pattern => "^\d{4}-\d{2}-\d{2} " 
        negate => true 
        what => previous 
        auto_flush_interval => 2 
        multiline_tag => "" 
    }
}

```

but if you are using filebeat you must do the multiline processing there. I don't run filebeat so I cannot help you with the syntax.

---

<div class="post-metadata">

**Author:** ![alex\_petrov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_petrov/32/105688_2.png) [@alex\_petrov](https://discuss.elastic.co/u/alex_petrov)\
**Post date:** [July 14, 2024, 12:46pm UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094/5 "2024-07-14T12:46:43Z")

</div>

may I use grok to parse first section of log till xml part starts for example :

> (?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:logLevel}._?hub\_(?[a-zA-Z0-9.\_-]+)._?%{UUID:correlationId}|body=%{GREEDYDATA:body}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 14, 2024, 1:09pm UTC](https://discuss.elastic.co/t/convert-embede-xml-in-log-to-json-in-logstash/363094/6 "2024-07-14T13:09:25Z")

</div>

Your example message does not have the hub\_ or UUID, but you could try something like

```
 grok { match => { "message" => "(?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}) %{LOGLEVEL:logLevel}.?(hub_(?<someField>[a-zA-Z0-9._-]+).?%{UUID:correlationId})?\|body=%{GREEDYDATA:body}" } }

```