# Error pushing logs to elasticsearch using kafka as input for logstash

**URL:** <https://discuss.elastic.co/t/error-pushing-logs-to-elasticsearch-using-kafka-as-input-for-logstash/214658>\
**Category:** Logstash\
**Created:** [January 10, 2020, 8:00pm UTC](https://discuss.elastic.co/t/error-pushing-logs-to-elasticsearch-using-kafka-as-input-for-logstash/214658 "2020-01-10T20:00:33Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Monu\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/monu_kumar/32/54790_2.png) [@Monu\_Kumar](https://discuss.elastic.co/u/Monu_Kumar)\
**Post date:** [January 10, 2020, 8:00pm UTC](https://discuss.elastic.co/t/error-pushing-logs-to-elasticsearch-using-kafka-as-input-for-logstash/214658/1 "2020-01-10T20:00:33Z")

</div>

Below is my logstash config file logstash.conf. Lotstash reads logs from Kafka topic 'my\_topic' and outputs it to elasticsearch index 'es-index'.

> input {  
> kafka {  
> bootstrap\_servers =\>"kafka1.xxx:9092,kafka2.xxx:9092,kafka3.xxx:9092"  
> topics =\> ["my\_topic"]  
> codec =\> "json"  
> group\_id =\> "logstashgroup"  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> ["es1.myhost:9200","es2.myhost:9200","es3.myhost:9200"]  
> user =\> "user123"  
> password =\> "password"  
> index =\> "es-index"  
> }  
> }  
> filter {  
> json {  
> source =\> "message"  
> skip\_on\_invalid\_json =\> true  
> }  
> }

It worked fine for a few months but recently it started throwing the following errors:

> [ERROR][logstash.outputs.elasticsearch] An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error\_message=\>"Could not read from stream: Corrupt GZIP trailer", :error\_class=\>"Manticore::StreamClosedException"

and,

> [FATAL][logstash.runner] An unexpected error occurred! {:error=\>org.apache.kafka.common.KafkaException: Received exception when fetching the next record from rumbl\_log-1. If needed, please seek past the record to continue consumption., :backtrace=\>["org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.fetchRecords(org/apache/kafka/clients/consumer/internals/Fetcher.java:1469)", "org.apache.kafka.clients.consumer.internals.Fetcher$PartitionRecords.access$1600(org/apache/kafka/clients/consumer/internals/Fetcher.java:1328)",

then,

> [ERROR][logstash.javapipeline] A plugin had an unrecoverable error. Will restart this plugin.  
> Plugin: \<LogStash::Inputs::Kafka codec=\>\<LogStash::Codecs::JSON id=\>"json\_9a7a9d96-d7be-4292-a3de-67f797d22ab5"

then,

> Error: Received exception when fetching the next record from my\_topic-1. If needed, please seek past the record to continue consumption.  
> Exception: Java::OrgApacheKafkaCommon::KafkaException

and then logstash stops

> [ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I'm not sure what's the problem. I couldn't find anything relevant to solve the error. .

---

<div class="post-metadata">

**Author:** ![Christophe\_Journel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christophe_journel/32/12534_2.png) [@Christophe\_Journel](https://discuss.elastic.co/u/Christophe_Journel)\
**Post date:** [January 30, 2020, 8:42am UTC](https://discuss.elastic.co/t/error-pushing-logs-to-elasticsearch-using-kafka-as-input-for-logstash/214658/2 "2020-01-30T08:42:47Z")

</div>

It seems that one of the kafka event in the partition "rumbl\_log-1" is corrupted.  
You should change the offset ( +1 ) to solve this issue - bypass this corrupted event

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2020, 8:48am UTC](https://discuss.elastic.co/t/error-pushing-logs-to-elasticsearch-using-kafka-as-input-for-logstash/214658/3 "2020-02-27T08:48:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.