Logstash-Kafka : each data pulled create new session on zookeeper


(Charlotte Dupont) #1

Hi,

I have an ELK suite with logstash pulling data from Kafka. Kafka is getting data from Fiware Orion-Cygnus (based on Apache Flume).

Everything is doing well for the 60 first data pushed. Then the connection with Cygnus is interrupted (error logs : "Connection reset by peer").

On the zookeeper side, I have the config of maximum 60 sessions per client. In the logs I see that zookeeper is disconnected because max of 60 sessios is reached.

Here is what happens :

2016-11-30 16:28:42,713 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /host:53190
2016-11-30 16:28:42,713 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@854] - Connection request from old client /host:53190; will be dropped if server is in r-o mode
2016-11-30 16:28:42,713 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900] - Client attempting to establish new session at /host:53190
2016-11-30 16:28:42,714 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x15844781a120334 with negotiated timeout 10000 for client /host:53190

2016-11-30 16:29:13,864 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /host:53192
2016-11-30 16:29:13,864 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@854] - Connection request from old client /host:53192; will be dropped if server is in r-o mode
2016-11-30 16:29:13,864 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900] - Client attempting to establish new session at /host:53192
2016-11-30 16:29:13,866 [myid:] - INFO [SyncThread:0:ZooKeeperServer@645] - Established session 0x15844781a120335 with negotiated timeout 10000 for client /host:53192

2016-11-30 16:29:13,902 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@192] - Accepted socket connection from /host:53194
2016-11-30 16:29:13,902 [myid:] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@854] - Connection request from old client /host:53194; will be dropped if server is in r-o mode
2016-11-30 16:29:13,902 [myid:] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@900] - Client attempting to establish new session at /host:53194

As you can see, same client (with new port) is establishing a new session when a new data is pulshed to kafka.

I assume it is coming from logstash as the Orion-Cygnus-Kafka config is working well without this problem. It only appears when connecting to ELK suite.

Do you have any idea how can I solve this problem ?
I found a very similar question here but couldn't manage to make it work by the answer.
I specified group name but doesn't change anything.

Thanks very much !


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.