LS I used both 2.4 and 5.0
FB is 5alpha6
FB is the following
for each file I have this path conf
fields_under_root: true
ignore_older: 10m
close_inactive: 2m
clean_inactive: 15m
#========================= Filebeat global options ============================
filebeat.spool_size: 10000
filebeat.idle_timeout: 10s
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["10.246.85.242:5044", "10.246.85.243:5044"]
template.name: "filebeat"
loadbalance: true
bulk_max_size: 1000
template.path: "filebeat.template.json"
template.overwrite: false
in LS i tried LOTS OF configuration 2, 3 and 4 instances of LS with different conf
the one that get me better results is the following, different configurations allowed me to pass a very smaller throughput (10% of the load) with this one I have almost 75%
path.data: /var/lib/logstash
pipeline.workers: 24
pipeline.output.workers: 1
pipeline.batch.size: 15000
path.config: /etc/logstash/conf.d
config.reload.automatic: true
config.reload.interval: 30
log.level: verbose
path.log: /var/log/logstash/logstash.log
with
-Xms12g
-Xmx12g
I have opened also this FileBeat EOF Error and this FileBeat file is falling under ignore older for other errors i am facing
hope you will help asap as my POC for my manager should be ready by next week