Base64 decoding & decompression of json

I have a JSON input containing a [message][message_json] field which is compressed & base64 encoded at the source.
This is the python code used to decode, decompress, and deserialize the field:

message['message_json'] = json.loads(zlib.decompress(base64.b64decode(message['message_json']), 15 + 32))

How can I replicate the same using logstash filter?

I tried to use logstash-filter-base64 to decode:

filter {
  json {
    source => "message"
    target => "message_deserialized"
  }

  base64 {
    field => "[message_deserialized][message_json]"
    action => "decode"
  }

But I'm getting "tags" => [[0] "_base64failure"] error.

Update:

I tried

ruby { 
    init => "require 'base64'"
    code => 'event.set("[message_deserialized][message_json_decoded]", Base64.decode64(event.get("[message_deserialized][message_json]")))' 
}

It works. Now I need help in just decompressing the [message_deserialized][message_json_decoded] field.

Update:

Successfully decompressed as well.

Final filter:

filter {
  json {
    source => "message"
    target => "message_deserialized"
  }

  ruby {
    init => "require 'base64'
             require 'zlib'
             require 'stringio'"
    code => 'event.set("[message_deserialized][message_json_decoded]", Zlib::GzipReader.new(StringIO.new(Base64.decode64(event.get("[message_deserialized][message_json]")))).read)' }

  json {
    source => "[message_deserialized][message_json_decoded]"
    target => "[message_deserialized][message_json_decoded_deserialized]"
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.