Unzip with ruby

I have flow python gzip script -> rabbitMQ -> logstash -> ES
If i subscribe to RMQ directly with python or ruby script - i can catch the message and decode it easily. But when it comes to logstash - ruby says that message is not in gzip format.

If I decode console-logged message with ruby - it works again. But inside logstash - nope.

For example: message inside console-log (between MESSAGE rows)

\u001F\x8B\b\u0000\xAD\xBE\u0018\\\u0002\xFF\x8B\xAEVJ\xCDM\xCC\xCCQ\xB2RP\xCAr\xC8\xD2K\xCE\xCFU\xD2QPJ\xCE\xCF+IL.\u0001\x89\x82\xB8y\x89\xB9\xA9 \xB6\u0017\x90.\u0006\t\u0014gg\xE6\xE4\u0014\u0003\x85\xA2\x95\xB4\r\u0015\xCC\xCD-,--\r\f\f̕b\x81\x92\xA9)\xA5@\x99j\xA5\xD2t\xB0\x82\x80ʒ\x8C\xFC<\xB0L\u0001D\xC49X)\xB6\xB6VGa\xE0\xEC\x8E\u0005\u0000\xEB*\xA5\xA3\xF8\u0000\u0000\u0000

Ruby exception occurred: not in gzip format

Logstash config:

input {
    rabbitmq {
        host => "localhost"
        port => 5672
        user => "guest"
        password => "guest"
        metadata_enabled => true
        queue => "kit_logstash"
        durable => true
        id => "rabbitmq_kit_event"
        exchange => "kit_local"
        codec => plain 
        type => "kitlog-rabbitmq"
    }
}

filter {
  if [type] == "kitlog-rabbitmq" {
    ruby {
      init => "
        require 'zlib'
        require 'stringio'
      "
      code => "
        body_decoded = event.get('message').to_s

        logger.info('===== MESSAGE =====')
        logger.info(body_decoded)
        logger.info('===== MESSAGE =====')

        sio = StringIO.new(body_decoded)
        gz = Zlib::GzipReader.new(sio)
        result = gz.read.to_s
        event.set('message', result)
      "
    }
  }
  json {
    source => "message"
    target => "message"
  }
  split {
    field => "message"
  }
}

output {
  stdout { codec => rubydebug }
  elasticsearch {
    hosts => "localhost"
    sniffing => false
    index => "kit_logs"
  }
}

Any ideas why ruby behaves like this?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.