I have flow python gzip script -> rabbitMQ -> logstash -> ES
If i subscribe to RMQ directly with python or ruby script - i can catch the message and decode it easily. But when it comes to logstash - ruby says that message is not in gzip format.
If I decode console-logged message with ruby - it works again. But inside logstash - nope.
For example: message inside console-log (between MESSAGE rows)
\u001F\x8B\b\u0000\xAD\xBE\u0018\\\u0002\xFF\x8B\xAEVJ\xCDM\xCC\xCCQ\xB2RP\xCAr\xC8\xD2K\xCE\xCFU\xD2QPJ\xCE\xCF+IL.\u0001\x89\x82\xB8y\x89\xB9\xA9 \xB6\u0017\x90.\u0006\t\u0014gg\xE6\xE4\u0014\u0003\x85\xA2\x95\xB4\r\u0015\xCC\xCD-,--\r\f\f̕b\x81\x92\xA9)\xA5@\x99j\xA5\xD2t\xB0\x82\x80ʒ\x8C\xFC<\xB0L\u0001D\xC49X)\xB6\xB6VGa\xE0\xEC\x8E\u0005\u0000\xEB*\xA5\xA3\xF8\u0000\u0000\u0000
Ruby exception occurred: not in gzip format
Logstash config:
input {
rabbitmq {
host => "localhost"
port => 5672
user => "guest"
password => "guest"
metadata_enabled => true
queue => "kit_logstash"
durable => true
id => "rabbitmq_kit_event"
exchange => "kit_local"
codec => plain
type => "kitlog-rabbitmq"
}
}
filter {
if [type] == "kitlog-rabbitmq" {
ruby {
init => "
require 'zlib'
require 'stringio'
"
code => "
body_decoded = event.get('message').to_s
logger.info('===== MESSAGE =====')
logger.info(body_decoded)
logger.info('===== MESSAGE =====')
sio = StringIO.new(body_decoded)
gz = Zlib::GzipReader.new(sio)
result = gz.read.to_s
event.set('message', result)
"
}
}
json {
source => "message"
target => "message"
}
split {
field => "message"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost"
sniffing => false
index => "kit_logs"
}
}
Any ideas why ruby behaves like this?