Unzip MQTT Message

Elastic Stack Logstash
1

Hi,

I'm using Mqtt plugin (Eclipse-Paho) to get messages from a MQTT topic, using the follow snippet:

input {
  mqtt {
	topic           => "/some/topic"
	host            => "${MQTT_HOST}"
	username        => "${MQTT_USERNAME}"
	password        => "${MQTT_PASSWORD}"
	client_id       => "client_id"
	clean_session   => false
	qos             => 1
}

}

After that a JSON filter its applied to transform the 'message' (from MQTT) to JSON.

	json {
  source    => "message"
}

But let's say my 'message' it's compressed using ZIP algorithm. Is that a way to uncompress and then apply the JSON filter mentioned before?

Thanks!

2

I'm not familiar with the MQTT protocol, but an initial glance at the protocol spec talks a fair bit about UTF-8 encoding and prohibiting the use of certain bytes (which could be problematic since compression algorithms don't like working in constrained byte ranges); does it support compressed fields or compressed payloads? If so, bringing that implementation up to the MQTT gem that does the bulk of the heavy lifting of Alexander Widar's Logstash MQTT Input may be your best bet.

If payloads are in fact constrained to UTF-8, a typical way to represent arbitrary binary data is by base64-encoding it, which carries a 4:3 overhead; depending on the size and repetition in your dataset, this may or may not be prohibitive compared to your compression savings.

I don't see an existing Logstash plugin for decompressing a String field; it would not be terribly hard to implement, but since Logstash's internal representation is typically a UTF-8 String, we would likely end up with something that took base64-encoded data, decoded it to binary compressed bytes, and then decompressed that.

The following quick-and-dirty script should work with the Ruby Filter Plugin:

3

Hi Ry,

Thanks very much for the help, that's exactly what I needed.
Yes, base64 can increase the final output so it's something we will have to think about.

Cheers.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.