CPU usage 100% when Elasticsearch is down

[fredrik.jonsson (Fredrik)]

I noticed that when Elasticsearch can't retrieve any more data the beats sending data went up to 100% CPU usage.
E.g. if the disc is full on the Elasticsearch server.
After clear up some space and "reset" Elasticsearch to be able to retrive data the CPU for the beats went down to normal behaviour.

It is never OK to use that much CPU and risk the other process running on the same machine.
Better loosing data than that the services stop working!

Is this a bug or can I config the beats in a better way?

[martinr_ubi (Martin R.)]

yes you can but can you share the config file of those beats and which beats they are first?
Versions also..

[fredrik.jonsson (Fredrik)]

The beats that were affected were, Packet, File and Metric

How do I paste in the config in a good way?
All the # in the config make some markup of the text.
Btw, the beats-config is the standard 6.7.1-config, we just changed hosts and path's to scan.

[willemdh]

Noticed the same thing on some client devices which are (not always) connected with a VPN. When the VPN connection is disabled or broken, CPU usage of metricbeat process seems to spike..

[fredrik.jonsson (Fredrik)]

Hi,
Any progress or news on this topic?

[steffens (Steffen Siering)]

Can you share configs, logs and CPU profiles? Please format logs and configs using the </>-Button.

All beats use exponential backoff on error. Backoff starts with 1s and goes up to 60s. The timeout is not reset if the connection was not re-established, getting you a retry every 1 minute. Filebeat event publish should be halted for this time, and only the file scanner looking for new/old files would be active.

What kind of data are you sending? How many files to you try to publish?

The bulk request is re-encoded to JSON after handling errors (as the error handler might drop events marked as invalid by Elasticsearch). This is when CPU can spike.