Kafka Integration Between Filebeat and Logstash

Elastic Stack Beats
1

Hi All,

I want to integrate KAFKA between Filebeat and Logstash .

My ELK stack is ready but to have more stability , i need this .

Please help me in configuring the same .
If anyone has DOCS for the same , Please do share .

2

Have you looked at the docs? What have you tried so far? What is not working?

3

Hi ,

Which docs you referring here ?

plz share link .

4

https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html

https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html

5

Ohh thanks , will go through them .

6

I read those docs but those are with logstash shipper and indexer .

do we have any docs with latest versions of Kafka , Filebeat and Logstash .

7

Hi ,

I have implemented Filebeat --- > kafka ---- > Logstash

but i am facing this issue while starting logstash

[root@vkafka02dsy logstash-6.0.0]# ./bin/logstash -f config/kafka.conf --config.reload.automatic
Sending Logstash's logs to /ELK/Logstach/logstash-6.0.0/logs which is now configured via log4j2.properties
[2017-12-12T10:38:38,971][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/ELK/Logstach/logstash-6.0.0/modules/fb_apache/configuration"}
[2017-12-12T10:38:38,975][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/ELK/Logstach/logstash-6.0.0/modules/netflow/configuration"}
[2017-12-12T10:38:39,175][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2017-12-12T10:38:39,399][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-12-12T10:38:39,887][ERROR][logstash.inputs.kafka ] Unknown setting 'zk_connect' for kafka
[2017-12-12T10:38:39,887][ERROR][logstash.inputs.kafka ] Unknown setting 'topic_id' for kafka
[2017-12-12T10:38:39,900][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/config/mixin.rb:114:in config_init'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/inputs/base.rb:62:ininitialize'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:152:in plugin'", "(eval):8:in'", "org/jruby/RubyKernel.java:994:in eval'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:82:ininitialize'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline.rb:215:in initialize'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/pipeline_action/create.rb:35:inexecute'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/agent.rb:90:in execute'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/runner.rb:362:inblock in execute'", "/ELK/Logstach/logstash-6.0.0/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

below is the way i am defining my input from kafka

input {
kafka {
zk_connect => "localhost:2181"
topic_id => "kafka"
}
}

is there any syntax error ? because i think the same .

i could not find any detailed configuration on Elastic and kafka sites .

8

with some modification getting different error ,

input {
kafka {
zk_connect => {"localhost:2181"}
topic_id => {"ITSB"}
}
}

[‎12/‎12/‎2017
5:16 PM] JAIN Shashank Pavan:

[2017-12-12T12:26:53,902][ERROR][logstash.agent
] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError",
:message=>"Expected one of #, => at line 3, column 36 (byte 54)
after input {\n kafka {\n zk_connect =>
{"localhost:2181"", :backtrace=>["/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/compiler.rb:42:in
compile_ast'", "/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/compiler.rb:50:incompile_imperative'",
"/ELK/Logstach/logstash-6.0.0/logstash-core/lib/logstash/compiler.rb:54:in
`compile_graph'", "/ELK/Logstach/logstash-6.0.0/logst

Can anyone please help on the same

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.