Hello everyone.
I have a Logstash pipeline reading from Oracle database a Blob column, named INPUT_MESSAGE.
In the pipepiline.yml I was using this configuration for the pipeline:
queue.type: memory
In the pipeline.config file, my input ruby filter correctly manages the Blob input in this way:
ruby {
init =>"require 'zlib'; require 'stringio';"
code =>"
message = event.get('input_message')
unless message.nil?
if (message.bytes[0] == 1 || message.bytes[0] == 4)
messageContentReader = Zlib::GzipReader.new(StringIO.new(message[1...message.length]))
messageContent = messageContentReader.read
messageContentReader.close
else
messageContent = message[1...message.length].to_s
end
event.set('input_message', messageContent)
event.set('input_message_size', messageContent.bytesize)
end
"
}
I've put a Logger Info to track value of the variable "message". It is, for example:
#<Sequel::SQL::Blob:0x7e0 bytes=89 start="\x01\x1F\x8B\b\x00\x00\x00\x00\x00\x04" end="\x01\x004\x88@\xA6Y\x00\x00\x00">
However, due to project requirements, I had to switch to a persisted queue configuration:
queue.type: persisted
In this case, the way of Logstash of storing the column value is different: the value of the variable message comes up to be:
"\u0001\u001F\u008B\b\u0000\u0000\u0000\u0000\u0000\u0004\u0000sv\nÖ\v\tÐstÔóÉOÏÌ\vI-.I-ÒKHUPÒ\u000Fµ\n\tpLÉÍÌË,.)J,É/R\u0002\n\u0006X\u0005$ª¨\u0094\e\u0014¥\u0080x¡\u008EØÔ8[ù*\u0001\u00004\u0088@¦Y\u0000\u0000\u0000"
So, in my input filter I receive the following error:
[ERROR][logstash.filters.ruby ] Ruby exception occurred: not in gzip format
Does anyone know how I can convert the message to the expected format, or acting on the way in wich Logstash is storing this field in the queue?
Thank you!