Kafka to Logstash

Hi Team ,

In Logstash I can see below error for pipeline having input as Kafka (Oracle Cloud)
It fails to connect and then discover .
Any idea on what needs to be checked. Once it is stable automatically, we gets data in Elastic.

Log Sample :
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][oic_xxx][0584xxxxxxx] [Consumer clientId=logstash-0, groupId=logstash] Group coordinator xxx.oci.oraclecloud.com:9092 (id: 2xxxx rack: null) is unavailable or invalid, will attempt rediscovery

[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][oic_xxx][058xxx] [Consumer clientId=logstash-0, groupId=logstash] Discovered group coordinator xxx.oci.oraclecloud.com:9092 (id: 2xxxx rack: null)

Those are not errors, they are informational messages.

It means the Logstash temporarily lost the connection with that Kafka broker and reconnected.

This is more a network/infrastructure issue.

What is the time difference between those two messages? You didn't share the full log line, so it is not possible to know.

Thanks Leandro Pereira.
Any idea on what can be done or what needs to be checked?
The time difference between messages are within milliseconds

Not sure, are you connecting to your Kafka using the public internet? From the log you shared it seems to be exposed to the internet: xxx.oci.oraclecloud.com.

Many cans cause network issues.

I do not see this as a problem since the connection it is reestablished after some miliseconds.

Is this happening frequently at the point of impact your ingestion?

We see this recently for every one hour and no data is ingested.

id: 2xxxx rack: null -- Does this means no data in topic ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.