# Kafka Integration Between Filebeat and Logstash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 8, 2017, 7:52am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799 "2017-12-08T07:52:36Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![SJN8](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Post date:** [December 8, 2017, 7:52am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/1 "2017-12-08T07:52:36Z")
Hi All, I want to integrate KAFKA between Filebeat and Logstash . My ELK stack is ready but to have more stability , i need this . Please help me in configuring the same . If anyone has DOCS for the same , Please do share . ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2017, 7:54am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/2 "2017-12-08T07:54:14Z")
Have you looked at the docs? What have you tried so far? What is not working? ---
**Author:** ![SJN8](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Post date:** [December 8, 2017, 8:10am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/3 "2017-12-08T08:10:18Z")
Hi , Which docs you referring here ? plz share link . ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 8, 2017, 8:15am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/4 "2017-12-08T08:15:16Z")
> **[Just Enough Kafka for the Elastic Stack, Part 1](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1)** > > This post talks about design considerations for integrating Kafka with the Elastic Stack. > **[Just Enough Kafka For The Elastic Stack, Part 2](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part2)** > > Are you running Apache Kafka with your Elastic Stack? Here's part 2 of the series which talks about operations and production deployment tips. [https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html](https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html) [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html) ---
**Author:** ![SJN8](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Post date:** [December 8, 2017, 8:16am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/5 "2017-12-08T08:16:08Z")
Ohh thanks , will go through them . ---
**Author:** ![SJN8](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Post date:** [December 8, 2017, 12:38pm UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/6 "2017-12-08T12:38:30Z")
I read those docs but those are with logstash shipper and indexer . do we have any docs with latest versions of Kafka , Filebeat and Logstash . ---
**Author:** ![SJN8](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Post date:** [December 12, 2017, 9:52am UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/7 "2017-12-12T09:52:49Z")
Hi , I have implemented Filebeat --- \> kafka ---- \> Logstash but i am facing this issue while starting logstash [root@vkafka02dsy logstash-6.0.0]# ./bin/logstash -f config/kafka.conf --config.reload.automatic Sending Logstash's logs to /ELK/Logstach/logstash-6.0.0/logs which is now configured via log4j2.properties [2017-12-12T10:38:38,971][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/ELK/Logstach/logstash-6.0.0/modules/fb\_apache/configuration"} [2017-12-12T10:38:38,975][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/ELK/Logstach/logstash-6.0.0/modules/netflow/configuration"} [2017-12-12T10:38:39,175][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2017-12-12T10:38:39,399][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} [2017-12-12T10:38:39,887][ERROR][logstash.inputs.kafka] Unknown setting 'zk\_connect' for kafka [2017-12-12T10:38:39,887][ERROR][logstash.inputs.kafka] Unknown setting 'topic\_id' for kafka [2017-12-12T10:38:39,900][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Something is wrong with your configuration.", :backtrace=\>["/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/config/mixin.rb:114:in `config_init'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/inputs/base.rb:62:in`initialize'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:152:in `plugin'", "(eval):8:in`'", "org/jruby/RubyKernel.java:994:in `eval'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:82:in`initialize'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:215:in `initialize'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline_action/create.rb:35:in`execute'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:335:in `block in converge_state'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:141:in`with\_pipelines'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:332:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in`each'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:319:in `converge_state'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:166:in`block in converge\_state\_and\_update'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:164:in`converge\_state\_and\_update'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:90:in `execute'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/runner.rb:362:in`block in execute'", "/ELK/Logstach/logstash-6.0.0/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]} below is the way i am defining my input from kafka input { kafka { zk\_connect =\> "localhost:2181" topic\_id =\> "kafka" } } is there any syntax error ? because i think the same . i could not find any detailed configuration on Elastic and kafka sites . ---
**Author:** ![SJN8](https://avatars.discourse-cdn.com/v4/letter/s/f08c70/32.png) [@SJN8](https://discuss.elastic.co/u/SJN8)\ **Post date:** [December 12, 2017, 12:07pm UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/8 "2017-12-12T12:07:48Z")
with some modification getting different error , input { kafka { zk\_connect =\> {"localhost:2181"} topic\_id =\> {"ITSB"} } } [‎12/‎12/‎2017 5:16 PM] JAIN Shashank Pavan: [2017-12-12T12:26:53,902][ERROR][logstash.agent ] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 3, column 36 (byte 54) after input {\n kafka {\n zk\_connect =\> {"localhost:2181"", :backtrace=\>["/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/compiler.rb:42:in `compile_ast'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/compiler.rb:50:in`compile\_imperative'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/compiler.rb:54:in `compile\_graph'", "/ELK/Logstach/logstash-6.0.0/logst Can anyone please help on the same ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 9, 2018, 12:08pm UTC](https://discuss.elastic.co/t/kafka-integration-between-filebeat-and-logstash/110799/9 "2018-01-09T12:08:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.