Filebeat low performance with multiple input files

Elastic Stack Beats
1

Hi everyone,
I have tried several configurations in the filebeat service but I cannot see a good performance. When I use a single input file we can see a high network throutput and logstash and elasticsearch do a good data processing.
However, when we use multiple intput files for filebeat the overall performance gets worse.
This is the parameter configuration we are using for multiple filebeat inputs:

filebeat.spool_size: 6144
queue.mem:
events: 131072

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["XXXX:5045", "XXXX:5045"]
loadbalance: true
worker: 4
pipelining: 0
ttl: 2s
bulk_max_size: 4096

Thanks for your help

2

Could you please share your full configuration formatted using </>?

3

This is my filebeat.yml

   #=========================== Filebeat inputs =============================
                filebeat.config.inputs:
                  enabled: true
                  path: inputs.d/*.yml
                  reload.enabled: true
                  reload.period: 10s
                #============================= Filebeat modules ===============================
                filebeat.config.modules:
                  path: ${path.config}/modules.d/*.yml
                  reload.enabled: false
                #================================ General =====================================
                  filebeat.spool_size: 6144
                queue.mem:
                  events: 131072
                #================================ Outputs =====================================
                #----------------------------- Logstash output --------------------------------
                output.logstash:
                  hosts: ["X.X.X.X:5045", "Y.Y.Y.Y:5045"]
                  loadbalance: true
                  worker: 4
                  pipelining: 0
                  ttl: 2s
                  bulk_max_size: 4096

And I have 12 .yml files under inputs.d folder with this structure:

- type: log
  enabled: true
  paths:
    - /folder/log/service/*.log
      fields_under_root: true
      fields: {type: service}

Thanks

4

The filebeat's version is 6.4.3

5

I don't see anything in your configuration which would lead to degraded performance.
What is your expectation regarding events/sec?
Could you please provide the debug logs of Filebeat? Do you see any errors in Logstash?

6

When I configure one input file for filebeat, I see up to 35Mbps (around 3300 paquets/sec) and when add more files I only get around 1Mbps (70 paquets/sec).

How do I get debug logs for filebeat? filebeat - e - d "*" ?

I don't get any errors in Logstash.

thanks

7

Any news about this bug?

thanks

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.