Kafka to Logstash

Elastic Stack Logstash
1

Hi Team ,

In Logstash I can see below error for pipeline having input as Kafka (Oracle Cloud)
It fails to connect and then discover .
Any idea on what needs to be checked. Once it is stable automatically, we gets data in Elastic.

Log Sample :
[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][oic_xxx][0584xxxxxxx] [Consumer clientId=logstash-0, groupId=logstash] Group coordinator xxx.oci.oraclecloud.com:9092 (id: 2xxxx rack: null) is unavailable or invalid, will attempt rediscovery

[INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator][oic_xxx][058xxx] [Consumer clientId=logstash-0, groupId=logstash] Discovered group coordinator xxx.oci.oraclecloud.com:9092 (id: 2xxxx rack: null)

2

Those are not errors, they are informational messages.

It means the Logstash temporarily lost the connection with that Kafka broker and reconnected.

This is more a network/infrastructure issue.

What is the time difference between those two messages? You didn't share the full log line, so it is not possible to know.

3

Thanks Leandro Pereira.
Any idea on what can be done or what needs to be checked?
The time difference between messages are within milliseconds

4

Not sure, are you connecting to your Kafka using the public internet? From the log you shared it seems to be exposed to the internet: xxx.oci.oraclecloud.com.

Many cans cause network issues.

I do not see this as a problem since the connection it is reestablished after some miliseconds.

Is this happening frequently at the point of impact your ingestion?

5

We see this recently for every one hour and no data is ingested.

id: 2xxxx rack: null -- Does this means no data in topic ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.