I am parsing Gihthub Log details from Logstash to Kafka and the logstash keeps throwing the below issue. After a certain time, I need to restart logstash to get rid of those warnings.
Once initiating the docker session, the logstash (Docker Containers) keeps throwing the below warn/err.
[2020-11-02T23:51:54,928][INFO ][logstash.outputs.kafka ][httpd][da57d4766911e4fad530f7fa630c3790519cb207ca20be19ea4a6ff35424df6d] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
[2020-11-02T23:51:54,930][WARN ][logstash.outputs.kafka ][httpd][da57d4766911e4fad530f7fa630c3790519cb207ca20be19ea4a6ff35424df6d] producer send failed {:exception=>Java::JavaUtilConcurrent::ExecutionException, :message=>"org.apache.kafka.common.errors.RecordTooLargeException: The message is 4369075 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration."}
[2020-11-02T23:51:54,930][INFO ][logstash.outputs.kafka ][httpd][da57d4766911e4fad530f7fa630c3790519cb207ca20be19ea4a6ff35424df6d] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
In the kafka compose files, i have mentioned the necessary settings for the size but it doesn't helps.
As I mentioned in the initial draft, logstash keeps throwing the warnings/infos and after a period of time, logstash stops accepting any inputs(http).
I still see the console log as below
[2020-11-03T16:35:08,762][INFO ][logstash.outputs.kafka ][httpd][da57d4766911e4fad530f7fa630c3790519cb207ca20be19ea4a6ff35424df6d] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
[2020-11-03T16:35:08,849][WARN ][logstash.outputs.kafka ][httpd][da57d4766911e4fad530f7fa630c3790519cb207ca20be19ea4a6ff35424df6d] producer send failed {:exception=>Java::JavaUtilConcurrent::ExecutionException, :message=>"org.apache.kafka.common.errors.RecordTooLargeException: The message is 3378103 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration."}
[2020-11-03T16:35:08,849][INFO ][logstash.outputs.kafka ][httpd][da57d4766911e4fad530f7fa630c3790519cb207ca20be19ea4a6ff35424df6d] Sending batch to Kafka failed. Will retry after a delay. {:batch_size=>1, :failures=>1, :sleep=>0.1}
After adding the max_request_size, the warnings have faded away.
Adding the updated output logstash phase here for future/others reference. Is there any other data parameter which I need to capture in the logstash output filter to avoid any other issues in future?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.