# Decode JSON hex field before push to ES

**URL:** <https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126>\
**Category:** Logstash\
**Created:** [October 23, 2020, 12:10pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126 "2020-10-23T12:10:17Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![cyberzlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cyberzlo/32/65490_2.png) [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\
**Post date:** [October 23, 2020, 12:10pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/1 "2020-10-23T12:10:17Z")

</div>

Hi, I have input data such as:

```auto
    {
      "timestamp": "1603363180424",
      "layers": {
          [.. some data ..]
        },
        "aaa": {
          [.. some data ..]
        },
        "bbb": {
          [.. some data ..]
        },
        "data": {
          "data_data_data": "3c:3f:78:6d:6c:20:76:65:72:73:69:6f:6e:3d:22:31:2e:30:22:20:65:6e:63:6f:64:69:6e:67:3d:22:75:74:66:2d:38:22:3f:3e:3c:73:6f:61:70:3a:45:6e:76:65:6c:6f:70:65:20:78:6d:6c:6e:73:3a:73:6f:61:70:3d:22:68:74:74:70:3a:2f:2f:77:77:77:2e:77:33:2e:6f:72:67:2f:32:30:30:33:2f:30:35:2f:73:6f:61:70:2d:65:6e:76:65:6c:6f:70:65:22:20:78:6d:6c:6e:73:3a:77:73:61:3d:22:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:34:2f:30:38:2f:61:64:64:72:65:73:73:69:6e:67:22:20:78:6d:6c:6e:73:3a:77:73:64:3d:22:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:35:2f:30:34:2f:64:69:73:63:6f:76:65:72:79:22:20:78:6d:6c:6e:73:3a:77:73:64:70:3d:22:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:36:2f:30:32:2f:64:65:76:70:72:6f:66:22:3e:3c:73:6f:61:70:3a:48:65:61:64:65:72:3e:3c:77:73:61:3a:54:6f:3e:75:72:6e:3a:73:63:68:65:6d:61:73:2d:78:6d:6c:73:6f:61:70:2d:6f:72:67:3a:77:73:3a:32:30:30:35:3a:30:34:3a:64:69:73:63:6f:76:65:72:79:3c:2f:77:73:61:3a:54:6f:3e:3c:77:73:61:3a:41:63:74:69:6f:6e:3e:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:35:2f:30:34:2f:64:69:73:63:6f:76:65:72:79:2f:50:72:6f:62:65:3c:2f:77:73:61:3a:41:63:74:69:6f:6e:3e:3c:77:73:61:3a:4d:65:73:73:61:67:65:49:44:3e:75:72:6e:3a:75:75:69:64:3a:36:37:31:65:36:35:30:33:2d:63:35:63:62:2d:34:66:35:38:2d:39:61:31:39:2d:65:62:62:32:30:37:64:36:63:36:62:37:3c:2f:77:73:61:3a:4d:65:73:73:61:67:65:49:44:3e:3c:2f:73:6f:61:70:3a:48:65:61:64:65:72:3e:3c:73:6f:61:70:3a:42:6f:64:79:3e:3c:77:73:64:3a:50:72:6f:62:65:3e:3c:77:73:64:3a:54:79:70:65:73:3e:77:73:64:70:3a:44:65:76:69:63:65:3c:2f:77:73:64:3a:54:79:70:65:73:3e:3c:2f:77:73:64:3a:50:72:6f:62:65:3e:3c:2f:73:6f:61:70:3a:42:6f:64:79:3e:3c:2f:73:6f:61:70:3a:45:6e:76:65:6c:6f:70:65:3e",
          "data_data_len": "624"
        }
      }
    }

```

How can I decode from hex this `data_data_data` to separated field? best would be addinional field `data_data_data_text` or just replace it. However any solution which will decode this will be great.

Looks like there is array `layers.data.data_data_data` in json hierarchy.

There is decoded text which I would like have also decoded in ES field of this event 🙂 - [https://tinyurl.com/y2ah8r9v](https://tinyurl.com/y2ah8r9v)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 23, 2020, 3:14pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/2 "2020-10-23T15:14:01Z")

</div>

> [@cyberzlo](#):
>
> How can I decode from hex this `data_data_data` to separated field?

I think you would have to use a ruby filter to do it.

---

<div class="post-metadata">

**Author:** ![cyberzlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cyberzlo/32/65490_2.png) [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\
**Post date:** [October 23, 2020, 3:27pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/3 "2020-10-23T15:27:18Z")

</div>

But do you know maybe how do it there? Because I have no idea 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 23, 2020, 3:43pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/4 "2020-10-23T15:43:28Z")

</div>

```
    mutate { gsub => ["someField", ":", ""] }
    ruby { code => 'event.set("anotherField", [event.get("someField")].pack("H*"))' }

```

will produce

```
"anotherField" => "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:wsa=\"http://schemas.xmlsoap.org/ws/2004/08/addressing\" xmlns:wsd=\"http://schemas.xmlsoap.org/ws/2005/04/discovery\" xmlns:wsdp=\"http://schemas.xmlsoap.org/ws/2006/02/devprof\"><soap:Header><wsa:To>urn:schemas-xmlsoap-org:ws:2005:04:discovery</wsa:To><wsa:Action>http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</wsa:Action><wsa:MessageID>urn:uuid:671e6503-c5cb-4f58-9a19-ebb207d6c6b7</wsa:MessageID></soap:Header><soap:Body><wsd:Probe><wsd:Types>wsdp:Device</wsd:Types></wsd:Probe></soap:Body></soap:Envelope>",
```

---

<div class="post-metadata">

**Author:** ![cyberzlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cyberzlo/32/65490_2.png) [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\
**Post date:** [October 23, 2020, 4:46pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/5 "2020-10-23T16:46:16Z")

</div>

Thanks but it doesn't work ☹

```auto
    mutate {
        gsub => ["data_data_data", ":", ""]
    }

    ruby {
        code => 'event.set("data_data_data_text", [event.get("data_data_data")].pack("H*"))'
    }

```

It returns always:

```auto
"data_data_data_text" => "",

```

BTW: How can I test it for output, not just config test if syntax is fine?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 23, 2020, 4:53pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/6 "2020-10-23T16:53:28Z")

</div>

You do not have a field called [data\_data\_data], you have a field called  
[data][data\_data\_data]

---

<div class="post-metadata">

**Author:** ![cyberzlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cyberzlo/32/65490_2.png) [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\
**Post date:** [October 23, 2020, 5:25pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/7 "2020-10-23T17:25:52Z")

</div>

Thanks! It almost work great, It is possible to create this new field at the end of my field list? Currently is on top in Kibana when I look for event details.

Probably most easy way will be just replace content of same field, it is possible? Because anyway I delete later this source field. So that will solve both problems, will not need to delate and I will have this field at the end 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 23, 2020, 5:56pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/8 "2020-10-23T17:56:22Z")

</div>

You can use event.set to overwrite the source field. Alternatively, you can event.set a field under [@metadata], which can be referenced in logstash but does not get written to elasticsearch. For example

```
ruby { code => 'event.set("[@metadata][xml]", [event.get("someField")].pack("H*"))' }
xml { source => "[@metadata][xml]" ... }
```

---

<div class="post-metadata">

**Author:** ![cyberzlo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cyberzlo/32/65490_2.png) [@cyberzlo](https://discuss.elastic.co/u/cyberzlo)\
**Post date:** [October 24, 2020, 4:47pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/9 "2020-10-24T16:47:27Z")

</div>

It works nice, but even when I replace/set value, then it is on top of all fields. it is possible to keep old position or place it on bottom or even sort by alphabet :)?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 24, 2020, 5:37pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/10 "2020-10-24T17:37:17Z")

</div>

fields in logstash and elasticsearch are not ordered. If you have a question about the display order of fields in kibana you should ask in that forum.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2020, 5:37pm UTC](https://discuss.elastic.co/t/decode-json-hex-field-before-push-to-es/253126/11 "2020-11-21T17:37:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.