Filebeat high cpu and bandwidth usage when ES is on flood stage

Elastic Stack Beats
1

When using Filebeat outputting to a single node Elasticsearch, when the cluster node enters flood stage due to high disk usage, the Filebeat clients will start consuming considerable percentage of CPU and network bandwidth. This also causes high CPU usage on ES node and it starts responding with 429 too many requests.

I see this on versions 8.16.0 and 8.18.0.

I tried setting a high value for backoff.init but it seems it had no effect.
Is there a way to rate limit filebeat for such scenario?

2

I've seem the use of Logstash with persistent queues as a mean for handling the back pressure. However, my understanding is that beats could better handle the 429 response from Elasticsearch.

3

You can try to enable the disk queue in beats to work in a similar way as the persistent queues in Logstash.

The main issue here is on Elasticsearch side that cannot write more data because it reached flood stage.

In this case it would be better to stop the ingestion to fix it than trying to find ways to deal with it on filebeat/logstash side.

I'm not sure if there are any settings regarding this, the backoff.init is related to errors while trying to connect to the endpoint, but in this case it can connect without any issue, Elasticsearch just reject the data.

Filebeat will try to send it until it is published.

4

Interesting that the "backpressure-sensitive protocol" probably is non existent when sending directly to Elasticsearch.
I'll research about the beats disk queue however I don't think it'll solve the bandwidth and ES CPU issue.
Thanks anyway.

5

The issue that you need to solve is Elasticsearch entering in a flood staging, this should be avoided.

Once it reaches flood stage it can impact ingestion no matter of the tool.

Logstash also can have similar issues increasing CPU usage because it start logging a lot of more errors, and this can increase IO on the system.

This may be the cause on Filebeat side, since it cannot write to Elasticsearch it starts logging more in disk, which increase IO usage.

But this is a consequence, the problem is Elasticsearch entering flood stage.

6

I understand your point. But the issue is: once in flood stage, for whatever reason, filebeat is unaware of it and causes a CPU and bandwidth spike. In such a scenario, and considering filebeat is running on multiple hosts, the only way to stop it is cutting network connectivity. It's the kind of remedy used on Logstash beats input: refusing connections.