Suggestion improving filebeat performance

[max_dodo (max)]

My ELK setup goes like :
Filebeats --> Logstash(ingest nodes) --> Elasticsearch ( master + Datanodes ) --> Kibana

Recently we are observing a huge amount of delay in logfile ingestion ( 4 ~ 5 hr) . From the capacity perspective we have added enough horsepower ( large-machines: 10+ ingestnodes, 15+ datanodes ) . Per day total log size reaches upto 900gb. Multiple applications generating huge amount of logs.

Note- on a daily basis around 100+ logfiles are generated on a single server, each of 500mb size.

Our filebeat configuration is as below. Please suggest if anything can be modified or added to take care of this performance issue.

---

filebeat.prospectors:

paths:
- /<log_path>/application*.log
fields:
level: debug
review: 1
json.keys_under_root: true
json.overwrite_keys: true
harvester_buffer_size: 16384
scan_frequency: 5s
document_type: <document_type_name>
registry_file: .filebeat
spool_size: 20480
tail_files: false
idle_timeout: 5s
input_type: log
max_backoff: 10s
max_bytes: 10485760

logging:
files:
keepfiles: 5
name: filebeat.log
path: /var/log/filebeat-logs
rotateeverybytes: 10485760
level: info
to_files: true
to_syslog: false

output.logstash:
hosts:
- "VIP-address:port"

[magnusbaeck (Magnus Bäck)]

Start by analyzing the whole pipeline and figuring out where the bottleneck is. Looking at the CPU load on the machines should give you an indication. Is the load distributed reasonably evenly across the Logstash hosts?

[max_dodo (max)]

Thanks for the response Magnus, Checked the whole ELK pipeline, there is no concern about the resource utilization on any of the nodes ( LS, DN, MN ) - CPU utilization hovers from 5-10% and MEM utilization is also normal.

We notices few things

1. there are multiple json parsefailures which we see in the logstash nodes
2. the number of logfiles opened by the filebeat is large on each client nodes ( Harvester started for file )

Can any of these cause the delay in ingestion of logfiles ?