# Kafka input and output plugin for logstash is not working
**URL:** https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271
**Category:** Logstash
**Created:** [February 19, 2016, 7:19pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271 "2016-02-19T19:19:25Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)
#### Post date: [February 19, 2016, 7:19pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/1 "2016-02-19T19:19:25Z")
Hello All,
I am using Logstash 2.1.1 and kafka version 2.10-0.8.2.1. I am getting following syntax error. What are the minumum required settings for kafka plugin.
Input.conf
input {
tcp {
port =\> 5010
#type =\> syslog
}
udp {
port =\> 5010
#type =\> syslog
}
}
output {
kafka {
zk\_connect =\> "10.195.115.15:2181"
topic\_id =\> "logstash-kafka"
}
}
Output.conf:
input {
kafka {
host =\> "10.195.115.15"
port =\> "9092"
type =\> "gaurav-type"
topic =\> "logstash-kafka"
message\_format =\> "json\_event"
}
}
output {
stdout {
codec =\> rubydebug
}
}
Error :
For input.conf
bash-3.2$ bin/logstash -f ../l1.conf --configtest
Unknown setting 'zk\_connect' for kafka {:level=\>:error}
Error: Something is wrong with your configuration.
For output.conf
bash-3.2$ bin/logstash -f ../l2.conf --configtest
Error: The setting `message_format` in plugin `kafka` is obsolete and is no longer available. Setting is no longer valid. If you have any questions about this, you are invited to visit [https://discuss.elastic.co/c/logstash](https://discuss.elastic.co/c/logstash) and ask.
I want to know simple config for kafka.
1: Listen syslong event at certain port
2: Put this syslog message to kafka topic
3: Pick message from Kafka topic
4: Parse it
5: Put it in elasticsearch
Thanks,
Gaurav
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [February 20, 2016, 2:13am UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/2 "2016-02-20T02:13:22Z")
Change message\_format in the Kafka output to codec =\> json {}
---
### Author:  [@gaurav1424](https://discuss.elastic.co/u/gaurav1424)
#### Post date: [February 22, 2016, 10:42pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/3 "2016-02-22T22:42:32Z")
Thanks Mate. but its not working.
Here is the output file
input {
tcp {
port =\> 5010
#type =\> syslog
}
udp {
port =\> 5010
#type =\> syslog
}
}
output {
kafka {
zk\_connect =\> "10.195.115.15:2181"
topic\_id =\> "logstash-kafka"
codec =\> json {}
}
}
bash-3.2$ bin/logstash -f ../l1.conf --configtest
Unknown setting 'zk\_connect' for kafka {:level=\>:error}
Error: Something is wrong with your configuration.
Thanks,
Gaurav
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [February 22, 2016, 11:29pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/4 "2016-02-22T23:29:59Z")
Make sure your parameters match the plugin. For example a Kafka output
needs bootstrap\_servers while an input needs zk\_connect.
[https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-kafka.html)
and
[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html)
---
### Author:  [@SKing](https://discuss.elastic.co/u/SKing)
#### Post date: [February 23, 2016, 3:18pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/5 "2016-02-23T15:18:28Z")
Thanks for reply Joe.
I'm actually at a similar stage. From looking at previous examples on the internet, it looks like there was a previous value brooker\_list where you would assign the address of your kafka queue. However, now it seems to have changed to bootstrap\_server.
Is that a correct interpretation?
I'm seeing other examples
kafka { host =\> "127.0.0.1" port =\> 9092 topic =\> "logstash" } [Reference:[https://gist.github.com/jeroenvandijk/4963802](https://gist.github.com/jeroenvandijk/4963802)]
output {
kafka {
:broker\_list =\> "[kafka1.mmlac.com:9092](http://kafka1.mmlac.com:9092)"
:topic\_id =\> "logstash"
:compression\_codec =\> "snappy"
:request\_required\_acks =\> 1
}
}
[Reference:[http://blog.mmlac.com/how-to-pre-process-logs-with-logstash/](http://blog.mmlac.com/how-to-pre-process-logs-with-logstash/)]
Kind regards,
Sarah
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [February 23, 2016, 5:31pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/6 "2016-02-23T17:31:18Z")
Yeah always refer to the plugin doc for your version. Generally changes are made on the logstash side for configuration keys to match what configuration values are used in Kafka.
So for example, Kafka Producers in 0.8 called it metadata.broker.list ([http://kafka.apache.org/082/documentation.html#producerconfigs](http://kafka.apache.org/082/documentation.html#producerconfigs)) while Kafka 0.9 producers call it broker.list ([http://kafka.apache.org/documentation.html#producerconfigs](http://kafka.apache.org/documentation.html#producerconfigs))
I know it is confusing but that is the reason for most of the changes.
---
### Author:  [@SKing](https://discuss.elastic.co/u/SKing)
#### Post date: [February 23, 2016, 5:57pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/7 "2016-02-23T17:57:31Z")
I guess the reason I am confused is when I look at the output plug information page for kafka [https://www.elastic.co/guide/en/logstash/2.2/plugins-outputs-kafka.html](https://www.elastic.co/guide/en/logstash/2.2/plugins-outputs-kafka.html), there is no longer an option for broker\_list.
Under bootstrap\_servers it says, "This is for bootstrapping and the producer will only use it for getting metadata (topics, partitions and replicas). The socket connections for sending the actual data will be established based on the broker information returned in the metadata. The format is host1:port1,host2:port2, and the list can be a subset of brokers or a VIP pointing to a subset of brokers."
However, does this mean that where I would previously have put the host and post opposite brooker\_list, I now use bootstrap\_servers?
Thanks,
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [February 23, 2016, 8:00pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/8 "2016-02-23T20:00:39Z")
Yes they are functionally equivalent but the new name more accurately describing the function of the parameter. Whereas previous broker\_list implies you supply a list of brokers to communicate with, the new one more accurately describes what is happening, ie you provide a list of servers to bootstrap from. In both situations the Kafka producer was just using the servers to get the latest metadata.
So yes, whatever your broker\_list was, you can use that value in bootstrap\_servers.
---
### Author:  [@SKing](https://discuss.elastic.co/u/SKing)
#### Post date: [February 24, 2016, 9:36am UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/9 "2016-02-24T09:36:30Z")
Thanks Joe, I appreciate your prompt feedback. It's been a big help.
Sarah
---
### Author:  [@hearvishwas](https://discuss.elastic.co/u/hearvishwas)
#### Post date: [August 16, 2016, 12:44pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/10 "2016-08-16T12:44:00Z")
logstash input plugin for kafka is not working for me: Here is my conf file:
input {
kafka { topic\_id =\> 'logstashlogs' }
}
output {
stdout{}
}
Nothing comes up to the screen, just lostash starts, no error, nothing is coming up to the screen. Kafka topic(logstashlogs) have data. I am able to inject data into kafka queue using logstash but not able to read it. Any suggestions?
---
### Author:  [@Joe\_Lawson](https://discuss.elastic.co/u/Joe_Lawson)
#### Post date: [August 16, 2016, 1:11pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/11 "2016-08-16T13:11:43Z")
You should start a new thread for this question.
Try running it with `--verbose` or `--debug` to see an error. [https://www.elastic.co/guide/en/logstash/current/command-line-flags.html](https://www.elastic.co/guide/en/logstash/current/command-line-flags.html)
---
### Author:  [@hearvishwas](https://discuss.elastic.co/u/hearvishwas)
#### Post date: [August 16, 2016, 1:35pm UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/12 "2016-08-16T13:35:27Z")
created new topic
"[Logstash Input Plugin for kafka 0.9 is not working](https://discuss.elastic.co/t/logstash-input-plugin-for-kafka-0-9-is-not-working/58146)"
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:43am UTC](https://discuss.elastic.co/t/kafka-input-and-output-plugin-for-logstash-is-not-working/42271/13 "2017-07-06T04:43:18Z")