Elastic Stack Version: 7.2.0
Kafka Version: 5.0.1
Using docker-compose to deploy Elastic Stack, with Kafka sitting on a remote system which is reachable via direct IP (192.168.0.2)
In the logstash.conf file we have:
input {
kafka {
bootstrap_servers => ["192.168.0.2:9092"]
topics => ["syslog"]
tags => ["from-kafka"]
}
}
...
I have confirmed using kafka-console-consumer that the endpoint is active and that the topic exists. However, when the logstash server is started the log entries show that logstash is trying to connect to localhost:
[2019-08-01T15:03:49,021][WARN ][org.apache.kafka.clients.NetworkClient] [Consumer clientId=logstash-0, groupId=logstash] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
I have tried every other possible configuration of kafka input such as:
kafka {
bootstrap_servers => "192.168.0.2:9092"
topics => ["syslog"]
tags => ["from-kafka"]
}
As shown in documentation, but the same error as above is shown. And:
kafka {
host => "192.168.0.2"
port => "9092"
topics => ["syslog"]
tags => ["from-kafka"]
}
But logstash errors out saying that host and port are unknown fields for kafka-input.
I am at a point where I do not believe that this can be a misconfiguration, as the config file is exactly as is described in the 7.2 documentation for kafka-input plugin.
Is there a new change in the kafka-input plugin that hasn't been properly documented in 7.2.0 documentation or is there an unknown bug in this new version? I haven't had a chance to downgrade to an older stack version but I will try asap to see if this is a versioning issue. Has anyone had any success getting a remote kafka-input setup working with ES 7.2.0 yet? I can get it working with localhost:9209 but we are in desperate need of having a decoupled Kafka-ES setup.
Thanks.