# How to read GZIP and encoding with UTF-8 logs from kafka topic through logstash pipeline **URL:** \ **Category:** Logstash\ **Created:** [December 21, 2023, 8:54am UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775 "2023-12-21T08:54:18Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![upreddy](https://avatars.discourse-cdn.com/v4/letter/u/f04885/32.png) [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Post date:** [December 21, 2023, 8:54am UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/1 "2023-12-21T08:54:19Z")
Hi All, Application team doing "GZIP and encoding with UTF-8" and sending their logs to kafka topics. Now i want to read those logs through logstash pipeline. Could you please guide me on this? Thanks ---
**Author:** ![upreddy](https://avatars.discourse-cdn.com/v4/letter/u/f04885/32.png) [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Post date:** [January 3, 2024, 8:55am UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/2 "2024-01-03T08:55:35Z")
Hi Team, Please guide me on the above topic? Thanks ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 3, 2024, 12:58pm UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/3 "2024-01-03T12:58:04Z")
What have you tried? You didn't provide any context. You need to use the Kafka input to read the messages from kafka, it will read compressed messages per default. ---
**Author:** ![upreddy](https://avatars.discourse-cdn.com/v4/letter/u/f04885/32.png) [@upreddy](https://discuss.elastic.co/u/upreddy)\ **Post date:** [January 3, 2024, 1:01pm UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/4 "2024-01-03T13:01:34Z")
Hi Leandrojmp, Here i am providing logstash configuration. input { kafka { bootstrap\_servers =\> "A,B,C" topics =\> "XX\_XX\_XX" group\_id =\> "XX\_XX\_XX" codec =\> gzip\_lines {charset =\> "UTF-8"} security\_protocol =\> "SASL\_SSL" sasl\_mechanism =\> "SCRAM-SHA-512" sasl\_jaas\_config =\> "org.apache.kafka.common.security.scram.ScramLoginModule required username='XX' password='XXXXXX';" ssl\_endpoint\_identification\_algorithm =\> "" } } filter { ruby { code =\> 'event.set("decoded", Base64.decode64(event.get("message")).force\_encoding("UTF-8"))' } } while using above configuration i am getting below error. [ERROR] 2024-01-03 05:22:51.005 [kafka-input-worker-logstash-0] Logstash - uncaught exception (in thread kafka-input-worker-logstash-0) org.jruby.exceptions.StandardError: (Error) not in gzip format at org.jruby.ext.zlib.JZlibRubyGzipReader.initialize(org/jruby/ext/zlib/JZlibRubyGzipReader.java:141) ~[jruby.jar:?] at org.jruby.ext.zlib.JZlibRubyGzipReader.new(org/jruby/ext/zlib/JZlibRubyGzipReader.java:85) ~[jruby.jar:?] at org.jruby.ext.zlib.JZlibRubyGzipReader.new(org/jruby/ext/zlib/JZlibRubyGzipReader.java:76) ~[jruby.jar:?] at usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_codec\_minus\_gzip\_lines\_minus\_3\_dot\_0\_dot\_4.lib.logstash.codecs.gzip\_lines.decode(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-codec-gzip\_lines-3.0.4/lib/logstash/codecs/gzip\_lines.rb:35) ~[?:?] at usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_integration\_minus\_kafka\_minus\_11\_dot\_2\_dot\_1\_minus\_java.lib.logstash.inputs.kafka.handle\_record(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-kafka-11.2.1-java/lib/logstash/inputs/kafka.rb:359) ~[?:?] at usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_integration\_minus\_kafka\_minus\_11\_dot\_2\_dot\_1\_minus\_java.lib.logstash.inputs.kafka.thread\_runner(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-kafka-11.2.1-java/lib/logstash/inputs/kafka.rb:329) ~[?:?] at usr.share.logstash.vendor.bundle.jruby.$2\_dot\_6\_dot\_0.gems.logstash\_minus\_integration\_minus\_kafka\_minus\_11\_dot\_2\_dot\_1\_minus\_java.lib.logstash.inputs.kafka.thread\_runner(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-integration-kafka-11.2.1-java/lib/logstash/inputs/kafka.rb:329) ~[?:?] ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 3, 2024, 1:18pm UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/5 "2024-01-03T13:18:04Z")
> [@upreddy](#): > > [ERROR] 2024-01-03 05:22:51.005 [kafka-input-worker-logstash-0] Logstash - uncaught exception (in thread kafka-input-worker-logstash-0) > org.jruby.exceptions.StandardError: (Error) not in gzip format This error means that the message is not on gzip format. Have you tried to remove this line? ```auto codec => gzip_lines {charset => "UTF-8"} ``` It is not clear what you mean with _GZIP and encoding with UTF-8_, the compression is done by Kafka normally and you do not need to do anything else as the messages would be uncompressed by default. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 3, 2024, 4:53pm UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/8 "2024-01-03T16:53:39Z")
How are your documents being written into Kafka? What tool is being used? It seems that you are compressing your documents and writing the compressed document into Kafka, is that right? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 31, 2024, 4:54pm UTC](https://discuss.elastic.co/t/how-to-read-gzip-and-encoding-with-utf-8-logs-from-kafka-topic-through-logstash-pipeline/349775/9 "2024-01-31T16:54:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.