# Unzip MQTT Message

**URL:** <https://discuss.elastic.co/t/unzip-mqtt-message/127855>\
**Category:** Logstash\
**Created:** [April 12, 2018, 5:10pm UTC](https://discuss.elastic.co/t/unzip-mqtt-message/127855 "2018-04-12T17:10:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![CasMeiron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casmeiron/32/24578_2.png) [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\
**Post date:** [April 12, 2018, 5:10pm UTC](https://discuss.elastic.co/t/unzip-mqtt-message/127855/1 "2018-04-12T17:10:48Z")

</div>

Hi,

I'm using Mqtt plugin (Eclipse-Paho) to get messages from a MQTT topic, using the follow snippet:

```
input {
  mqtt {
	topic => "/some/topic"
	host => "${MQTT_HOST}"
	username => "${MQTT_USERNAME}"
	password => "${MQTT_PASSWORD}"
	client_id => "client_id"
	clean_session => false
	qos => 1
}

```

}

After that a JSON filter its applied to transform the 'message' (from MQTT) to JSON.

```
	json {
  source => "message"
}

```

But let's say my 'message' it's compressed using ZIP algorithm. Is that a way to uncompress and then apply the JSON filter mentioned before?

Thanks!

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [April 13, 2018, 1:42am UTC](https://discuss.elastic.co/t/unzip-mqtt-message/127855/2 "2018-04-13T01:42:03Z")

</div>

I'm not familiar with the MQTT protocol, but an initial glance at the protocol spec talks a fair bit about UTF-8 encoding and prohibiting the use of certain bytes (which could be problematic since compression algorithms don't like working in constrained byte ranges); does it support compressed fields or compressed payloads? If so, bringing that implementation up to the [MQTT gem](https://github.com/njh/ruby-mqtt) that does the bulk of the heavy lifting of [Alexander Widar's Logstash MQTT Input](https://github.com/widarlein/logstash-input-mqtt) may be your best bet.

If payloads _are_ in fact constrained to UTF-8, a typical way to represent arbitrary binary data is by base64-encoding it, which carries a 4:3 overhead; depending on the size and repetition in your dataset, this may or may not be prohibitive compared to your compression savings.

I don't see an existing Logstash plugin for decompressing a String field; it would not be terribly hard to implement, but since Logstash's internal representation is typically a UTF-8 String, we would likely end up with something that took base64-encoded data, decoded it to binary compressed bytes, and then decompressed that.

The following quick-and-dirty script should work with the [Ruby Filter Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html):

> <https://gist.github.com/yaauie/0b769d8f6d034080e3603f48606913d0>

---

<div class="post-metadata">

**Author:** ![CasMeiron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casmeiron/32/24578_2.png) [@CasMeiron](https://discuss.elastic.co/u/CasMeiron)\
**Post date:** [April 13, 2018, 1:10pm UTC](https://discuss.elastic.co/t/unzip-mqtt-message/127855/3 "2018-04-13T13:10:41Z")

</div>

Hi Ry,

Thanks very much for the help, that's exactly what I needed.  
Yes, base64 can increase the final output so it's something we will have to think about.

Cheers.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 11, 2018, 1:10pm UTC](https://discuss.elastic.co/t/unzip-mqtt-message/127855/4 "2018-05-11T13:10:45Z")

</div>

Closed on May 11, 2018, 1:10pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.