We configured logstash to use "kafka" as input and "elasticsearch" as output. But it seems that the logstash instance can not consume the messages from kafka.
Kindly check the logstash version and its configuration as below:
[root@elk logstash-1.5]# logstash --version
[root@elk logstash-1.5]# cat /etc/logstash/central.conf
zk_connect => "hdp4:2181"
topic_id => "test"
cluster => "logstash"
As you can see, the connection with zookeeper has established:
[root@elk logstash-1.5]# netstat -an | grep 2181
tcp 0 0 ::ffff:10.28.0.233:54260 ::ffff:10.80.3.234:2181 ESTABLISHED
[root@elk logstash-1.5]# ping hdp4
PING hdp4 (10.80.3.234) 56(84) bytes of data.
64 bytes from hdp4 (10.80.3.234): icmp_seq=1 ttl=63 time=0.729 ms
64 bytes from hdp4 (10.80.3.234): icmp_seq=2 ttl=63 time=1.08 ms
And we can also assure that there are messages in kafka can be consumed:
[root@hdp6 bin]# ./kafka-console-consumer.sh --zookeeper 10.80.3.234:2181 --topic test --from-beginning
May 13 14:25:54 lek-aix61 auth|security:err|error sshd: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
May 13 14:25:56 lek-aix61 auth|security:info sshd: Accepted password for root from 10.28.0.233 port 58526 ssh2
May 13 14:29:20 lek-aix61 auth|security:info sshd: Received disconnect from 10.28.0.233: 11: disconnected by user
Logged into the zookeeper server, and we found that there was no "offset" sub-directory generated for logstash:
[zk: localhost:2181(CONNECTED) 31] ls /consumers/console-consumer-28776
[offsets, owners, ids]
[zk: localhost:2181(CONNECTED) 32] ls /consumers/logstash
Thanks a lot for your support in advance.