Filebeat performance tuning

[ashwani]

Hi,
We recently upgraded filebeat (1.0.1 to 5.0.1), logstash (2.1.1 to 2.4.0) versions on our rpm based 8 core cpu with 14 GB RAM in our filebeat -> logstash -> Elastic Search -> Kibana application. All configurations are bare minimum in filebeat and logstash. Earlier we were getting throughput of around 2000 tps for 60 MB logs. After the upgrade, tps has been dropped to 1200-1500 tps. Do I need to make any other configuration change in the new versions to boost up the performance

[ruflin]

Can you share your filebeat and logstash configuration file? It could be that in filebeat some default configs change, and if you don't have it in your config file, it will just take the new defaults.

[ashwani]

Thanks for your reply.

Logstash configuration :

input {
file{
path => "/usr/local/apps/monitoring/monitor.log"
add_field => {"iname" => "monitor"}
 codec => "json"
}
 beats {
    port => 5044
    codec => "json"
   congestion_threshold =>30
   add_field => {"iname" => "bee"}
  }
beats{
        port => 5045
        codec => "json"
        add_field => {"iname" => "mis"}
        }
}
filter {
 mutate {
        remove_field => [ "fields","input_type" , "offset","host","beat","n","h","p","v","l" ]
        }
  date {
    match => [ "t", "ISO8601","UNIX_MS" ]
  }
  if [iname] == "mis" {
        geoip {
                source=> ip
                }
  }
}
output {
if [iname] == "bee" {
  elasticsearch { hosts => ["SERVER-IP"]
                document_id => "%{[j][cid]}-%{[j][msgid]}"
                index => "logstash-%{[iname]}-%{+YYYY.MM.dd}"
                workers => 8
  }
} else {
elasticsearch { hosts => ["SERVER-IP"]
                index => "logstash-%{[iname]}-%{+YYYY.MM.dd}"
                workers => 8

                }
}

}

Filebeat configuration :

filebeat.prospectors:
	paths:
        - /usr/local/apps/myAppp/log/log.*
	input_type: log
	ignore_older: 24h
	
output.logstash:
	hosts: ["SERVER-IP"]
	worker: 4
	
logging.level: warning
logging.to_files: false
logging.to_syslog: false

Please let me know if I need to share the configurations we were using in earlier versions. Will share them too.

[ruflin]

Did you use different configs in earlier version? If that is the case, it would be helpful to also share these. The first step would be to figure out where the slow down happened. As you have FB -> LS -> ES lots of different areas can affect performance.

Mainly speaking on the filebeat side, it can be helpful for performance to increase filebeat.spool_size as you get larger batches. As the logstash automatically increases / decreases the window size, it's worth to have a look at your logs to see if you see something suspicious there, for example failed acked events which would mean filebeat slows down sending.

For the beats-input make sure you have the most recent version of the plugin. Some more performance improvements were added to the 5.0 release of Logstash, so if you can it would be worth to upgrade to 5.0.

How many filebeat instances do you have sending to Logstash? How many logstash instances do you have?