Hi all,
We got some little issues in production around Logstash and we are a bit worried for the future, we would like to know the community's opinion on this. Let me explain:
We have ran Logstash in production so far as a docker container with default JVM options (1GB heap). We started recently to use Winlogbeat directly on Windows endpoints (~ 80 endpoints to begin) to gather windows events in ECS format. Our winlogbeat configuration output an average of 230 events per second for all the endpoints. Our logstash instance is doing much more and have to handle an average of 1000 events per second.
Everything ran smooth until we started to listen for beats connections in logstash. Indeed, it ran out of memory. Increasing the java heap space to 8GB fixes the issue once for all. The thing is, we are going to continue the deployment of Winlogbeat to 12.000 endpoints. With our latest issue we had in production, we are a bit concerned about the memory that would require. We are a bit scared if Logstash would be able to sustain so many connections simultaneously according to the maximum memory we can provide: 32GB.
My first feelings would have said the memory required will be about the internal queues containing the events. But, I would be interested to know what could be the memory overhead of using so many beats connections in Logstash, or this can be considered out of concerns but the events filling-in internal queues !? Note that:
- Using a single Winlogbeat instance on a Windows Events Collector is unfortunately not an option
- We run a reverse proxy with mutual TLS in front of Logstash
What you guys think about it?
Thx
input {
beats {
port => 5044
}
}
filter {
ruby {
path => "/usr/share/logstash/logstash_ruby_scripts/endpoint_meta.rb"
script_params => {
api_key_field => "[meta][key]"
master_url => "http://api-center/endpoint/api_key/validate"
ca_path => "/etc/ssl/certs/ca-certificates.crt"
cache_duration => 300
failed_cache_duration => 60
}
}
ruby {
code => "event.set('[@metadata][ingest_date]', Time.now.strftime('%Y-%m-%d_%H_%M'))"
}
mutate {
copy => { "[meta][deployment_name]" => "[@metadata][filename]" }
}
mutate {
gsub => [
"[@metadata][filename]", "[^A-Za-z0-9]", ""
]
}
}
output {
file {
path => "/data/host_logs/winlogbeat/%{[@metadata][filename]}_windows-%{[@metadata][ingest_date]}.json"
}
}
I don't know if it's safe to run with 12K connections and if the log volume only matters. If a java queue is created per connection, it could be (I guess) also an attention point. I don't know about the implementation actually.
