Filebeat slow registry issues

Elastic Stack Beats
1

Hello Team,
I have an Opensearch cluster with 10 Opensearch Indexers running on Kubernetes.
I have Logstash in the middle to process all the logs. Who send the logs is Filebeat, but the delay in the log ingestion on Opensearch is huge.
Checking all, I can see the Filebeat is slow on harvesting the log file.

This is my configuration:

filebeat:
  inputs:
	- type: log
	  paths:
		- "/var/log/log_file.json"
	  close_removed: true
	  clean_inactive: 7h
	  clean_removed: true
	  ignore_older: 6h
output:
  logstash:
	  hosts: ["logstash-oss:5000"]
	  bulk_max_size: 4096
	  worker: 15
	  compression_level: 3
	  pipelining: 2
queue:
  mem:
	flush:
	  min_events: 2048
	  timeout: 1s

I added the the clean and flush options to solve the issue but it does not worked.
When I check the Filebeat registry for all the Filebeat instances, I see this:

> for filebeat in $filebeats; do echo $filebeat; echo $(($(($(date +%s) - $(k-alias -n $ENV exec $filebeat -- bash -c "find /var/lib/filebeat/registry/filebeat/ -type f -regex '.*\/[0-9]+.json' -exec jq '.[-1].timestamp[1]' {} \;")))/60)); done
filebeat1-0
62
filebeat-0
1662036
filebeat-1
153
filebeat-10
201
filebeat-11
221
filebeat-2
1661991
filebeat-3
1662036
filebeat-4
1662036
filebeat-5
1661991
filebeat-6
177
filebeat-7
114
filebeat-8
332
filebeat-9
53

And these numbers keeps increasing.

I tried:

  • Increasing the number of Logstash instances to help ingest the logs
  • Increasing the number of Opensearch nodes.
  • Restarting Filebeat services
  • Restarting Filebeat pods
  • Touching the Filebeat options

But nothing solved the issue. Can you help me with this?

2

Hello and welcome,

Most of the time the bootleneck is in the destination, in this case it would be in Opensearch.

You need to check if you already tuned opensearch and improved what you can, but Opensearch is not supported here, you need to check on the Opensearch forum.

Most of the tunning settings for Elasticsearch would work on Opensearch, but not everything as it is not the same time anymore.

If your Opensearch cluster cannot index fast enough, then it will tell Logstash to backoff, which will then tell Filebeat to backoff, so increasing the number of Logstash instances will not help.

3

Thank you @leandrojmp
Your recommendations make total sense.
We have several OpenSearch clusters in the same condition, and this is the only one that is slowing the ingestion. Do you have any tunning recommendation?
For the moment we added one more node to the cluster.
Thanks again.

4

I do not use Opensearch, only Elasticsearch.

As mentioned the tunning tips for Elasticsearch may also apply to Opensearch, but you need to check it yourself, Opensearch is not supported here.

1 Like