Logstash is not sending the data to elasticsearch

[sauravsuman689 (Saurav Suman)]

Hi Team,

My dataflow is as below :

filebeat -> kafka -> logstash -> es -> kibana

Messages are coming to Kafka topic but I don't see logstash is processing anything and sending to es cluster.

Below is the warning messages I see in logstash logs.
 
**
[2021-01-27T11:19:08,908][WARN ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] [Consumer clientId=logstash-2, groupId=prod_app] This member will leave the group because consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.

**

I tried tuning logstash config with the below parameter but still getting warnings.

input {
  kafka {
    bootstrap_servers => "xxxxxxxxxx:9092"
    topics =>  ["prod_external", "prod_internal"]
    group_id =>  "prod_app"
    consumer_threads => 5
    codec => "json"
    heartbeat_interval_ms => "1000"
    max_poll_records => "100"
    poll_timeout_ms => "10000"
    request_timeout_ms => "120000"
    session_timeout_ms => "130000"

Please advise.

[sauravsuman689 (Saurav Suman)]

Hi Team,

Can someone please advise on how to fix the above issue?

Regards,
Saurav Suman

[tatdat (Dat Pham Tat)]

I can't give you the solution extracly but i think you can try to debug.

filebeat -> kafka -> logstash -> es -> kibana

1. Check new event in Kafka. If kafka still got new event from filebeat, jump to (4), or not, jump to to (2)
2. Kafka didn't new event, check filebeat, (run filebeat with debug to make sure filebeat is working as well filebeat -e -d "*" ). If FB worked, jump to check kafka (3)
3. Check Kafka. Did Kafka still worked ?
4. FB -> Kafka is worked, check on logstash, run logstahs with debug or verbose mode.

I think you can find problem. Good luck!