# Logstash with Kafka compression and decompression **URL:** \ **Category:** Logstash\ **Created:** [April 10, 2018, 4:18am UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412 "2018-04-10T04:18:50Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![hat\_trick](https://avatars.discourse-cdn.com/v4/letter/h/898d66/32.png) [@hat\_trick](https://discuss.elastic.co/u/hat_trick)\ **Post date:** [April 10, 2018, 4:18am UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/1 "2018-04-10T04:18:50Z")
Hi, I'm new to Elastic and very interested in this pipeline: Data Sources --\>LogStash --\> Kafka --\>LogStash --\> ElasticSearch, where the first LS specifies gzip data compression with Kafka output plugin and the second LS enriches data with filter plugins. I assume a gzip codec plugin is required on the second LS in order to process the data, does that mean decompression happens on the second LS? or on the final ES? Also, where does the compression actually happen, on the first LS or Kafka? Thanks in advance! ---
**Author:** ![hat\_trick](https://avatars.discourse-cdn.com/v4/letter/h/898d66/32.png) [@hat\_trick](https://discuss.elastic.co/u/hat_trick)\ **Post date:** [April 10, 2018, 5:57pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/2 "2018-04-10T17:57:30Z")
Anyone have experience on logstash kafka compressed data transfer? Thanks!!! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 10, 2018, 6:10pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/3 "2018-04-10T18:10:33Z")
> [@hat\_trick](#): > > I assume a gzip codec plugin is required on the second LS in order to process the data No it is not. I have logstash reading from kafka, discarding 99% of the data and writing with gzip compression to another kafka instance. Another logstash instance reads that topic and it does not specify compression on the input. ---
**Author:** ![hat\_trick](https://avatars.discourse-cdn.com/v4/letter/h/898d66/32.png) [@hat\_trick](https://discuss.elastic.co/u/hat_trick)\ **Post date:** [April 10, 2018, 7:10pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/4 "2018-04-10T19:10:13Z")
Thanks Badger, looks like you also have logstash -\> kafka -\> logstash. I want to do some data processing work on the second logstash, in this case, I assume a gzip codec is required. Do you have the same data processing work running on the second logstash? Do you happen to know where the compression happens, on the first logstash or on kafka ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 10, 2018, 7:17pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/5 "2018-04-10T19:17:38Z")
You do not require a gzip codec on the second logstash instance. The kafka message header indicates whether the message is compressed, so the input plugin will know whether to decompress. I believe the kafka producer (i.e. logstash) is expected to do the compression but I am not certain. ---
**Author:** ![hat\_trick](https://avatars.discourse-cdn.com/v4/letter/h/898d66/32.png) [@hat\_trick](https://discuss.elastic.co/u/hat_trick)\ **Post date:** [April 10, 2018, 8:05pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/6 "2018-04-10T20:05:47Z")
Cool...This is interesting, then I guess elasticsearch can handle gzip somehow ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 8, 2018, 8:05pm UTC](https://discuss.elastic.co/t/logstash-with-kafka-compression-and-decompression/127412/7 "2018-05-08T20:05:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.