Decode JSON hex field before push to ES

[cyberzlo]

Hi, I have input data such as:

    {
      "timestamp": "1603363180424",
      "layers": {
          [.. some data ..]
        },
        "aaa": {
          [.. some data ..]
        },
        "bbb": {
          [.. some data ..]
        },
        "data": {
          "data_data_data": "3c:3f:78:6d:6c:20:76:65:72:73:69:6f:6e:3d:22:31:2e:30:22:20:65:6e:63:6f:64:69:6e:67:3d:22:75:74:66:2d:38:22:3f:3e:3c:73:6f:61:70:3a:45:6e:76:65:6c:6f:70:65:20:78:6d:6c:6e:73:3a:73:6f:61:70:3d:22:68:74:74:70:3a:2f:2f:77:77:77:2e:77:33:2e:6f:72:67:2f:32:30:30:33:2f:30:35:2f:73:6f:61:70:2d:65:6e:76:65:6c:6f:70:65:22:20:78:6d:6c:6e:73:3a:77:73:61:3d:22:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:34:2f:30:38:2f:61:64:64:72:65:73:73:69:6e:67:22:20:78:6d:6c:6e:73:3a:77:73:64:3d:22:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:35:2f:30:34:2f:64:69:73:63:6f:76:65:72:79:22:20:78:6d:6c:6e:73:3a:77:73:64:70:3d:22:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:36:2f:30:32:2f:64:65:76:70:72:6f:66:22:3e:3c:73:6f:61:70:3a:48:65:61:64:65:72:3e:3c:77:73:61:3a:54:6f:3e:75:72:6e:3a:73:63:68:65:6d:61:73:2d:78:6d:6c:73:6f:61:70:2d:6f:72:67:3a:77:73:3a:32:30:30:35:3a:30:34:3a:64:69:73:63:6f:76:65:72:79:3c:2f:77:73:61:3a:54:6f:3e:3c:77:73:61:3a:41:63:74:69:6f:6e:3e:68:74:74:70:3a:2f:2f:73:63:68:65:6d:61:73:2e:78:6d:6c:73:6f:61:70:2e:6f:72:67:2f:77:73:2f:32:30:30:35:2f:30:34:2f:64:69:73:63:6f:76:65:72:79:2f:50:72:6f:62:65:3c:2f:77:73:61:3a:41:63:74:69:6f:6e:3e:3c:77:73:61:3a:4d:65:73:73:61:67:65:49:44:3e:75:72:6e:3a:75:75:69:64:3a:36:37:31:65:36:35:30:33:2d:63:35:63:62:2d:34:66:35:38:2d:39:61:31:39:2d:65:62:62:32:30:37:64:36:63:36:62:37:3c:2f:77:73:61:3a:4d:65:73:73:61:67:65:49:44:3e:3c:2f:73:6f:61:70:3a:48:65:61:64:65:72:3e:3c:73:6f:61:70:3a:42:6f:64:79:3e:3c:77:73:64:3a:50:72:6f:62:65:3e:3c:77:73:64:3a:54:79:70:65:73:3e:77:73:64:70:3a:44:65:76:69:63:65:3c:2f:77:73:64:3a:54:79:70:65:73:3e:3c:2f:77:73:64:3a:50:72:6f:62:65:3e:3c:2f:73:6f:61:70:3a:42:6f:64:79:3e:3c:2f:73:6f:61:70:3a:45:6e:76:65:6c:6f:70:65:3e",
          "data_data_len": "624"
        }
      }
    }

How can I decode from hex this data_data_data to separated field? best would be addinional field data_data_data_text or just replace it. However any solution which will decode this will be great.

Looks like there is array layers.data.data_data_data in json hierarchy.

There is decoded text which I would like have also decoded in ES field of this event - https://tinyurl.com/y2ah8r9v

[Badger]

@cyberzlo

How can I decode from hex this data_data_data to separated field?

I think you would have to use a ruby filter to do it.

[cyberzlo]

But do you know maybe how do it there? Because I have no idea

[Badger]

    mutate { gsub => [ "someField", ":", "" ] }
    ruby { code => 'event.set("anotherField", [event.get("someField")].pack("H*"))' }

will produce

"anotherField" => "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://www.w3.org/2003/05/soap-envelope\" xmlns:wsa=\"http://schemas.xmlsoap.org/ws/2004/08/addressing\" xmlns:wsd=\"http://schemas.xmlsoap.org/ws/2005/04/discovery\" xmlns:wsdp=\"http://schemas.xmlsoap.org/ws/2006/02/devprof\"><soap:Header><wsa:To>urn:schemas-xmlsoap-org:ws:2005:04:discovery</wsa:To><wsa:Action>http://schemas.xmlsoap.org/ws/2005/04/discovery/Probe</wsa:Action><wsa:MessageID>urn:uuid:671e6503-c5cb-4f58-9a19-ebb207d6c6b7</wsa:MessageID></soap:Header><soap:Body><wsd:Probe><wsd:Types>wsdp:Device</wsd:Types></wsd:Probe></soap:Body></soap:Envelope>",

[cyberzlo]

Thanks but it doesn't work

    mutate {
        gsub => [ "data_data_data", ":", "" ]
    }

    ruby {
        code => 'event.set("data_data_data_text", [event.get("data_data_data")].pack("H*"))'
    }

It returns always:

"data_data_data_text" => "",

BTW: How can I test it for output, not just config test if syntax is fine?

[Badger]

You do not have a field called [data_data_data], you have a field called
[data][data_data_data]

[cyberzlo]

Thanks! It almost work great, It is possible to create this new field at the end of my field list? Currently is on top in Kibana when I look for event details.

Probably most easy way will be just replace content of same field, it is possible? Because anyway I delete later this source field. So that will solve both problems, will not need to delate and I will have this field at the end

[Badger]

You can use event.set to overwrite the source field. Alternatively, you can event.set a field under [@metadata], which can be referenced in logstash but does not get written to elasticsearch. For example

ruby { code => 'event.set("[@metadata][xml]", [event.get("someField")].pack("H*"))' }
xml { source => "[@metadata][xml]" ... }

[cyberzlo]

It works nice, but even when I replace/set value, then it is on top of all fields. it is possible to keep old position or place it on bottom or even sort by alphabet :)?

[Badger]

fields in logstash and elasticsearch are not ordered. If you have a question about the display order of fields in kibana you should ask in that forum.