How can I temporary disable filebeat in a Kafka downtime?

[elasticheart]

Hi,

I have a number of filebeats, sending data to Kafka, with number worker set to 2. I have Kafka downtimes, and when it becomes available, Filebeats sends a huge data as a bulk. Refer the below trace;

[2017-02-03 09:13:54,813] WARN Unexpected error from /192.168.0.1; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 842019127 larger than 104857600)
        at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:91)
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:71)
        at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:154)
        at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:135)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:323)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
        at kafka.network.Processor.poll(SocketServer.scala:472)
        at kafka.network.Processor.run(SocketServer.scala:412)
        at java.lang.Thread.run(Thread.java:745)

I would like to temporarily disable Filebeats / discard the filebeat outputs in a Kafka downtime. How can I do that?

Thank you.

[warkolm (Mark Walkom)]

You cannot, it should realise kafka is unavailable and stop processing though.

[elasticheart]

Ok.. so will reachable_only: true solve my problem?

[steffens (Steffen Siering)]

when using reachable_only: true, please also take potential resource usage into account. See note.

Also consider enabling quotas in kafka: http://kafka.apache.org/documentation/#design_quotas