# Decompress a gzip compressed string in logstash and push to es

**URL:** <https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720>\
**Category:** Logstash\
**Tags:** docker\
**Created:** [March 15, 2023, 5:48am UTC](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720 "2023-03-15T05:48:36Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![shdasgupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shdasgupta/32/118450_2.png) [@shdasgupta](https://discuss.elastic.co/u/shdasgupta)\
**Post date:** [March 15, 2023, 5:48am UTC](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720/1 "2023-03-15T05:48:36Z")

</div>

Team, I am trying to **decompress a gzip compressed data** using logstash - am not able to figure out how to do this.

**Input json:**  
_(this is fed through a file in this example. In actual, it is consuming a kafka message which looks similar)_

```auto
{
  "id": "238dbf3e-34d6-4a8c-a7cf-39a091642754",
  "version": 1,
  "ttl": 2592000,
  "createdDate": 1678501916554,
  "modifiedDate": 1678501930238,
  "payload": {
    "compressedPayload": "H4sIAAAAAAAACqtWykvMTVVSsFJQysrPyFNIyU9V0lFQykxRslIwNAACICcxJaUotbgYpKhaQSk5s6QSKKkUnJin4JVfDFaenF+aV1IEFg4NdlSqrQUAsk/IEFcAAAA=",
    "compresisonType": "GZIP"
  },
  "type": "single"
}

```

**Desired output:**  
Decompressed content inside payload.compressedPayload (the below is decompressed using a online gzip decompresser)  
ie

```auto
{
  "name": "john doe",
  "id": 10000,
  "address": {
    "city": "San Jose",
    "country": "USA"
  }
}

```

My logstash **pipeline config** looks like this

```auto
input {
  file {
        path=> "/usr/share/logstash/sample-kafka-pipeline.json"
        start_position =>"beginning"
        sincedb_path => "/dev/null"
  }
}

filter {
    json {
        source => "[message][payload][compressedPayload]"
        target => "[message]"
    }
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => "localhost:9200"
        index => "test-docker-logstash"
    }
}

```

I know my filter doesn't look right. But I would like to know what is the best way to decompress the gzipped string in my input json.

Any help is appreciated. Thanks in advance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 15, 2023, 4:05pm UTC](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720/2 "2023-03-15T16:05:53Z")

</div>

You will have to use a ruby filter.

```
input { generator { count => 1 lines => ['{ "payload": { "compressedPayload": "H4sIAAAAAAAACqtWykvMTVVSsFJQysrPyFNIyU9V0lFQykxRslIwNAACICcxJaUotbgYpKhaQSk5s6QSKKkUnJin4JVfDFaenF+aV1IEFg4NdlSqrQUAsk/IEFcAAAA=" } }'] codec => json } }

output { stdout { codec => rubydebug { metadata => false } } }
filter {
    ruby {
        code => '
            begin
                p = event.get("[payload][compressedPayload]")
                gzipData = Base64.decode64(p)
                sio = StringIO.new(gzipData)
                gz = Zlib::GzipReader.new(sio, encoding: Encoding::ASCII_8BIT)
                u = gz.read
                event.set("someField", u)
            ensure
                gz&.close
            end
        '
    }
}

```

which will produce

```
 "someField" => "{\"name\" : \"john doe\", \"id\": 10000, \"address\" : { \"city\": \"San Jose\", \"country\": \"USA\"}}",

```

---

<div class="post-metadata">

**Author:** ![shdasgupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shdasgupta/32/118450_2.png) [@shdasgupta](https://discuss.elastic.co/u/shdasgupta)\
**Post date:** [March 20, 2023, 2:18am UTC](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720/3 "2023-03-20T02:18:45Z")

</div>

Thank you so much @Badger for the prompt reply.  
This worked great, after a minor tweak to fit it to my need. The filter was flawless.  
Being still new to ELK, this answer opened other possibilities for me.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2023, 2:19am UTC](https://discuss.elastic.co/t/decompress-a-gzip-compressed-string-in-logstash-and-push-to-es/327720/4 "2023-04-17T02:19:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.