Hello,
We use Kafka input and output plugins with Logstash in our log pipeline. We are using 2.3.1 version of Logstash and Elasticsearch. I see that there is now Kafka 0.9 producer/consumer support in LS 5.0. Any chance you would backport that to 2.3.1? Or is there a way I can retrofit to use that plugin with my Logstash 2.3.1? We are using Kafka 0.9, and are guaranteed not to have any 0.8 brokers, so backward compatibility is not an issue for me.
We have secured with the pipeline from filebeat to Logstash. And also from Logstash to Elasticsearch. Would like to do this as well to get end-to-end SSL.
EDIT: of course, minutes after posting, I run into this:
So modifying my question to: when is this becoming GA?
@allenmchan@suyograo I tried switching to the recently GA'd 0.9 client based Kafka plugins as mentioned above, and getting some errors during conf validation. Could you please help? Here are the details:
Thanks @allenmchan. Looks like for input conf it is called "topics" whereas for output conf it is still called topic_id. Would be great if they can be the same. But at least it does not complain anymore.
Also, looks like reset_beginning is no longer supported config. What (if any) is the equivalent in 3.0.x please?
yes, this is probably what I am looking for thanks! I will read up a little more on implications of not setting this to one of the options provided (reset_beginning used to have a "false"). In 0.9 client we have to explicitly set to one of 4 values, and that is not a bad thing.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.