# Parse a json file that includes an xml

**URL:** <https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312>\
**Category:** Logstash\
**Created:** [October 24, 2022, 12:36pm UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312 "2022-10-24T12:36:03Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alexandros](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@Alexandros](https://discuss.elastic.co/u/Alexandros)\
**Post date:** [October 24, 2022, 12:36pm UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/1 "2022-10-24T12:36:03Z")

</div>

Hello all,

I want to send the following json document to elasticsearch through logstash.

```auto
     "short_message": "<?xml version="1.0" encoding="utf-8"?>
      <ImportMessageBase xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      <Version>1.0.0</Version>
      <Direction>Inbound</Direction>
      <Topic>test</Topic>
      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>
      <Entity>AVIS_LAGOS2BSM</Entity>
      <Company>07SC</Company>
      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8QVZfQVZJ=</Data>",
	  "host": "mock-service-6596db7999-m8r7c"} ' > /dev/udp/127.0.0.1/5041

```

I **dont** want to parse the "short\_message" field and disolve its xml elements.

I just want to see in kibana the field: " **short\_message**" that will include my **whole xml** message and the " **host**": "mock-service-6596db7999-m8r7c" as a separate field in the same indexed document.

In order to do so i send through my Ubuntu cmd the following command:

```auto
echo '{ "short_message": "<?xml version="1.0" encoding="utf-8"?>
      <ImportMessageBase xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
      <Version>1.0.0</Version>
      <Direction>Inbound</Direction>
      <Topic>test</Topic>
      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>
      <Entity>AVIS_LAGOS2BSM</Entity>
      <Company>07SC</Company>
      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8QVZfQVZJ=</Data>",
	  "host": "mock-service-6596db7999-m8r7c"} ' > /dev/udp/127.0.0.1/5041

```

The json document reaches Elasticsearch but with a grok parse failure message.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c72ba989f1fede9e2ccf2f6e94d733b995bbe28a.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0c8e76ab98f5fa1713b31e88d2b578ca35c0f0dc.png)

My pipeline configuration is as follows:

```auto
input {

    gelf {

          codec => multiline {

            pattern => "<?xml version"

            what => "next"

           }

          port_udp => 5041

          use_udp => true

          id => "gelf"

   }

}

output {

    elasticsearch {

        hosts => ["http://localhost:9200"]

        index => "alex"

    }

}

```

Any possible solution concerning how to change my pipeline configuration file?

Thank you

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [October 25, 2022, 7:10am UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/2 "2022-10-25T07:10:52Z")

</div>

Hello Alexandros,

I think your json document really is malformed:

> { "short\_message": "\<?xml version="1.0" encoding="utf-8"?\>

You have to escape the quotes in the XML:

> { "short\_message": "\<?xml version=\"1.0\" encoding=\"utf-8\"?\>

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![Alexandros](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@Alexandros](https://discuss.elastic.co/u/Alexandros)\
**Post date:** [October 29, 2022, 11:53am UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/3 "2022-10-29T11:53:40Z")

</div>

Hello @ [Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)

Thank you very much for your reply.

I changed the xml part of my json as you advised to the following form:

```auto
echo '{ "short_message": "<?xml version=/"1.0/" encoding=/"utf-8/"?>
      <ImportMessageBase xmlns:xsi=/"http://www.w3.org/2001/XMLSchema-instance/" xmlns:xsd=/"http://www.w3.org/2001/XMLSchema/">
      <Version>1.0.0</Version>
      <Direction>Inbound</Direction>
      <Topic>test</Topic>
      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>
      <Entity>AVIS_LAGOS2BSM</Entity>
      <Company>07SC</Company>
      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8QVZfQVZJ=</Data>",
	  "host": "mock-service-6596db7999-m8r7c"} ' > /dev/udp/127.0.0.1/5041

```

Nevertheless, my document is now spitted in 2 parts as the image show below with the (\t,\n, \ ) characters that i dont want to see them in kibana UI:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/9/8941643b52f0f9bb30ed7731339e907c452a9973.png)

I also receive json grok parse failure as the image show below:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/8/a8e7b0f3d8bae381e49124ed70f63f52f5983b5a.png)

The relative log from logstash mentions the following:

```auto
[2022-10-29T14:35:07,679][ERROR][logstash.inputs.gelf][main][gelf] JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Unexpected character ('1' (code 49)): was expecting comma to separate Object entries

 at [Source: (byte[])"{ "short_message": "<?xml version=/"1.0/" encoding=/"utf-8/"?>

      <ImportMessageBase xmlns:xsi=/"http://www.w3.org/2001/XMLSchema-instance/" xmlns:xsd=/"http://www.w3.org/2001/XMLSchema/">

      <Version>1.0.0</Version>

      <Direction>Inbound</Direction>

      <Topic>test</Topic>

      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>

      <Entity>AVIS_LAGOS2BSM</Entity>

      <Company>07SC</Company>

      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8"[truncated 19 bytes]; line: 1, column: 38]>, :data=>"\"{ \\\"short_message\\\": \\\"<?xml version=/\\\"1.0/\\\" encoding=/\\\"utf-8/\>

< expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')

 at [Source: (byte[])"\u0009 "host": "mock-service-6596db7999-m8r7c"}

"; line: 1, column: 11]>, :data=>"\"\\t \\\"host\\\": \\\"mock-service-6596db7999-m8r7c\\\"} \\n\""}

```

Any ideas on how to solve that one?

Thank you a lot in advance

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 29, 2022, 1:27pm UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/4 "2022-10-29T13:27:36Z")

</div>

> [@Alexandros](#):
>
> `echo '{ "short_message": "<?xml version=/"1.0/"`

These should be `\"`, not `/"`, throughout.

---

<div class="post-metadata">

**Author:** ![Alexandros](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@Alexandros](https://discuss.elastic.co/u/Alexandros)\
**Post date:** [November 7, 2022, 7:39am UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/5 "2022-11-07T07:39:44Z")

</div>

Hello @Badger , @ [Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)

Thank you and sorry for my silly mistake.

I changed the message to the following format:

```auto
echo '{ "short_message": "<?xml version=\"1.0\" encoding=\"utf-8\"?>

      <ImportMessageBase xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">

      <Version>1.0.0</Version>

      <Direction>Inbound</Direction>

      <Topic>test</Topic>

      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>

      <Entity>AVIS_LAGOS2BSM</Entity>

      <Company>07SC</Company>

      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8QVZfQVZJ=</Data>",

          "host": "mock-service-6596db7999-m8r7c"} ' > /dev/udp/127.0.0.1/5041

```

Nevertheless, my message is splitted in 2 parts as the image show below:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/5/c5d7d623cca92588d76fc50794fd6d307ecb27f2.png)

More specifically, i receive again json grok parse failure and in the message section i see the escape " \ " symbol that i have inserted:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8b4519248e0793324de58255e304c761d08bbf1f.png)

The relative logstash error log is exactly the following:

```auto
[2022-11-07T09:27:28,393][ERROR][logstash.inputs.gelf][main][gelf] JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value

 at [Source: (byte[])"{ "short_message": "<?xml version=\"1.0\" encoding=\"utf-8\"?>

      <ImportMessageBase xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">

      <Version>1.0.0</Version>

      <Direction>Inbound</Direction>

      <Topic>test</Topic>

      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>

      <Entity>AVIS_LAGOS2BSM</Entity>

      <Company>07SC</Company>

      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8"[truncated 19 bytes]; line: 1, column: 64]>, :data=>"\"{ \\\"short_message\\\": \\\"<?xml version=\\\\\\\"1.0\\\\\\\" encoding=\\\\\\>

[2022-11-07T09:27:29,967][ERROR][logstash.inputs.gelf][main][gelf] JSON parse failure. Falling back to plain-text {:error=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expec>

 at [Source: (byte[])"\u0009 "host": "mock-service-6596db7999-m8r7c"}

"; line: 1, column: 11]>, :data=>"\"\\t \\\"host\\\": \\\"mock-service-6596db7999-m8r7c\\\"} \\n\""}

```

Any ideas on that one??

Thank you for your time.

Best regards,  
Alexandros

---

<div class="post-metadata">

**Author:** ![Alexandros](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@Alexandros](https://discuss.elastic.co/u/Alexandros)\
**Post date:** [November 16, 2022, 7:19am UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/6 "2022-11-16T07:19:58Z")

</div>

Hello @Badger , @ [Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)

Any possible feedback on that one?

Thank you in advance,

Best regards,  
Alexandros

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 16, 2022, 5:03pm UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/7 "2022-11-16T17:03:25Z")

</div>

> [@Alexandros](#):
>
> LogStash::Json::ParserError: Illegal unquoted character ((CTRL-CHAR, code 10)): has to be escaped using backslash to be included in string value

It is telling you that you cannot have an unquoted newline within a JSON object.

---

<div class="post-metadata">

**Author:** ![Alexandros](https://avatars.discourse-cdn.com/v4/letter/a/ecc23a/32.png) [@Alexandros](https://discuss.elastic.co/u/Alexandros)\
**Post date:** [November 18, 2022, 8:40am UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/8 "2022-11-18T08:40:06Z")

</div>

Hello @Badger,

Thank you, So in order to understand since what i want to send is the following message:

```auto
echo '{ "short_message": "<?xml version=\"1.0\" encoding=\"utf-8\"?>

      <ImportMessageBase xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\">

      <Version>1.0.0</Version>

      <Direction>Inbound</Direction>

      <Topic>test</Topic>

      <ConversationId>{EA73CD72-96DC-EC2C-E053-3EA0010A1319}</ConversationId>

      <Entity>AVIS_LAGOS2BSM</Entity>

      <Company>07SC</Company>

      <Data>PD94bWwgdmVyc2lvbj0iMS4wIj8+CjxBVklTX0xBR09TMkJTTT4KICA8QVZfQVZJ=</Data>",

          "host": "mock-service-6596db7999-m8r7c"} ' > /dev/udp/127.0.0.1/5041

```

How should i transform it so that i dont have that previous error anymore?

Best regards,  
Alexandros

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2022, 8:40am UTC](https://discuss.elastic.co/t/parse-a-json-file-that-includes-an-xml/317312/9 "2022-12-16T08:40:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.