Random number of messages dropping on logstash import

[Paul_Shriner]

How do I discover the LS backlog?

I just found this thread, while older, I have been playing with batch size (100 - 3000 )

Losing messages during high traffic rate

I am doing log analysis using Filebeat (1.2) -> logstash(2.3) -> Elasticsearch (2.3) I have 4 filebeat instances, 4 logstash instances (6 cores each) , and Elasticsearch cluster (8 cores, 64G RAM) of 2 nodes In Filebeat.yml logstash setting is as such: filebeat: prospectors: - paths: - /var/log/filebeat//.json encoding: utf-8 input_type: log ignore_older: 10m scan_frequency: 1s exclude_lines: ["^$"] spool_size: 3072 registry_file: .filebeat output: logstash: enabled: true hos…

That seems to be a dead end as well.