Logstash and Kafka, nothing happening and nothing in the logs

Elastic Stack Logstash
1

Hi everyone,

I have a setup that looks like this:

Capture.PNG477×568 8.28 KB

(The data nodes also have the master role)

With Rsyslog used as a lightweight forwarder between the rest of my network and my ELK setup, storing the logs in Kafka so my Logstash cluster can pull them at its pace.

The Rsyslog/Kafka part works well, with logs being pushed to the desired topic, but Logstash doesn't seem to be pulling anything.

Here is my logstash config:

input {
  kafka {
    topic_id => "logstash-logs-apps"
    group_id => "logstash-apps-cluster"
    zk_connect => "XX.XX.XX.XX:2181"
    type => "logs-apps"
    codec => "line"
  }

  ##
  # a (winlog)beat input working fine here
  ##
}

output {
  if [type] == "logs-apps" {
    elasticsearch {
      hosts => ["XX.XX.XX.XX","XX.XX.XX.XX"]
      index => "logs-apps-%{+YYYY.MM.dd}"
    }
  }

  ##
  # an elasticsearch output for the beat input working fine here
  ##
}

All the machines are Ubuntu 14.04 VMs, with the latest Elasticsearch and Logstash versions installed (2.2.x), and Kafka 0.9.0.0
The connections between my logstash servers and my kafka server are established (as shown with "netstat -uta") but it appears as if Logstash wasn't doing anything regarding this input and no logs are generated by Logstash, ES, or Kafka.

Any thoughts on this issue would be highly appreciated!

Thanks,

Nicolas

2

Make sure your brokers are running 0.8.X mode. Otherwise you need to run Kafka 0.8.X as the 0.9 support isn't present in Logstash yet.

1 Like
3

Hi Joe,

Thanks for the reply!
Following your advice, I am now using kafka 0.8.2.2, and have the following logs from zookeeper when my logstash cluster is connecting to zookeeper:

[2016-03-09 16:39:46,985] INFO Accepted socket connection from /:50595 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-03-09 16:39:46,988] INFO Client attempting to establish new session at /:50595 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-09 16:39:46,990] INFO Established session 0x1535bfd09ed0007 with negotiated timeout 6000 for client /10.200.40.30:50595 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-09 16:39:47,108] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0007 type:create cxid:0x1 zxid:0x29 txntype:-1 reqpath:n/a Error Path:/consumers/logstash-apps-cluster/ids Error:KeeperErrorCode = NoNode for /consumers/logstash-apps-cluster/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:39:47,116] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0007 type:create cxid:0x2 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/consumers/logstash-apps-cluster Error:KeeperErrorCode = NoNode for /consumers/logstash-apps-cluster (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:39:47,419] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0007 type:create cxid:0x14 zxid:0x2e txntype:-1 reqpath:n/a Error Path:/consumers/logstash-apps-cluster/owners/logstash-logs-apps Error:KeeperErrorCode = NoNode for /consumers/logstash-apps-cluster/owners/logstash-logs-apps (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:39:47,426] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0007 type:create cxid:0x15 zxid:0x2f txntype:-1 reqpath:n/a Error Path:/consumers/logstash-apps-cluster/owners Error:KeeperErrorCode = NoNode for /consumers/logstash-apps-cluster/owners (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:39:48,848] INFO Accepted socket connection from /:37598 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-03-09 16:39:48,850] INFO Client attempting to establish new session at /:37598 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-09 16:39:48,852] INFO Established session 0x1535bfd09ed0008 with negotiated timeout 6000 for client /10.200.40.31:37598 (org.apache.zookeeper.server.ZooKeeperServer)

(continues in the next post, reached the characters limit...)

Thanks

4

But when I run the command "bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic logstash-logs-apps", I can see the logs from Rsyslog being pushed to the topic and have the following logs in zookeeper:

[2016-03-09 16:41:35,703] INFO Accepted socket connection from /127.0.0.1:50229 (org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-03-09 16:41:35,711] INFO Client attempting to establish new session at /127.0.0.1:50229 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-09 16:41:35,713] INFO Established session 0x1535bfd09ed0009 with negotiated timeout 6000 for client /127.0.0.1:50229 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-03-09 16:41:35,828] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0009 type:create cxid:0x2 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-9646/ids Error:KeeperErrorCode = NoNode for /consumers/console-consumer-9646/ids (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:41:35,837] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0009 type:create cxid:0x3 zxid:0x39 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-9646 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-9646 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:41:36,196] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0009 type:create cxid:0x19 zxid:0x3d txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-9646/owners/logstash-logs-apps Error:KeeperErrorCode = NoNode for /consumers/console-consumer-9646/owners/logstash-logs-apps (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:41:36,197] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0009 type:create cxid:0x1a zxid:0x3e txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-9646/owners Error:KeeperErrorCode = NoNode for /consumers/console-consumer-9646/owners (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:42:08,585] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0009 type:setData cxid:0x24 zxid:0x42 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-9646/offsets/logstash-logs-apps/0 Error:KeeperErrorCode = NoNode for /consumers/console-consumer-9646/offsets/logstash-logs-apps/0 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:42:08,587] INFO Got user-level KeeperException when processing sessionid:0x1535bfd09ed0009 type:create cxid:0x25 zxid:0x43 txntype:-1 reqpath:n/a Error Path:/consumers/console-consumer-9646/offsets Error:KeeperErrorCode = NoNode for /consumers/console-consumer-9646/offsets (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:42:08,593] INFO Processed session termination for sessionid: 0x1535bfd09ed0009 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-03-09 16:42:08,594] INFO Closed socket connection for client /127.0.0.1:50229 which had sessionid 0x1535bfd09ed0009 (org.apache.zookeeper.server.NIOServerCnxn)

Thanks again

5

Did you clean out zookeeper prior to downgrading? I'd expect some incompatibles in the structure. If you can't nuke and pave, cleaning out the consumers path on ZK.

6

Just nuked it to be safe and get the very same result...

Advice?

Thanks,
Nicolas