Kafka Output Recurring I/O Timeout

[Or_Arnon]

Hi,

We're seeing Filebeat getting:

	kafka/log.go:53	producer/broker/1036 state change to [closing] because write tcp 10.200.1.158:49334->10.200.3.121:9092: i/o timeout

This causes Filebeat to re-connect to Kafka brokers and reduce speed.
Anyone with a solution or a way to debug?

[jsoriano (Jaime Soriano)]

Hi @Or_Arnon,

Is any event successfully sent or all connections to Kafka fail?

[Or_Arnon]

All events are sent. However, we do get these timeouts which forces Filebeat to re-discover all partitions.

[Noy_Meir]

this is our complete config:

filebeat.inputs:

• type: log
  enabled: true
  paths:
  • /ephemeral/log/haproxy/haproxy*
    exclude_lines: ['^C']
    fields_under_root: true
    json.keys_under_root: true
    json.add_error_key: true
    json.message_key: message
    json.ignore_decoding_error: true
    scan_frequency: 1s
    multiline.pattern: '^{'
    multiline.match: after

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true

filebeat.config.inputs:
reload.enabled: true
reload.period: 60s

output.kafka:
enabled: true
hosts: [ .... ]
worker: 1
version: 0.11.0.0
compression: snappy
bulk_max_size: 4098
required_acks: 0
message.max.bytes: 20000000
client_id: "gateway_haproxy"
topic: "test_filebeat"
partition.round_robin:
group_events: 1
reachable_only: true

logging.level: info

http.enabled: true

xpack.monitoring:
enabled: true
elasticsearch:
hosts: [ ... ]