Hi,
our setup (pilot): 10 neflow exporters (cca 10k events/sec) to logstash with netflow modul to elastic. In the beginning (after restart logstash) everything is ok but after 10 min input is droping to 1k events/sec (kibana netflow discover attach). We were playing with this parameters to even get 10k per sec in start (bellow). Please help/advice.
ELK: 1 node vmware redhat 7.6, 36cores, 128GB ram, jvm 32 GB
logstash:
pipeline.batch.size: 3000
pipeline.batch.delay: 5
pipeline.workers: 96
modules:
- name: netflow
var.input.udp.port: 2056
var.input.udp.buffer_size: 655360
var.input.udp.queue_size: 40000
var.input.udp.workers: 36
var.input.udp.receive_buffer_bytes: 134217728
var.elasticsearch.hosts: http://127.0.0.1:9200
var.elasticsearch.ssl.enabled: false
var.kibana.host: 127.0.0.1:5601
var.kibana.scheme: http
var.kibana.ssl.enabled: false
var.kibana.ssl.verification_mode: disable
var.output.elasticsearch.pool_max: 80000
var.output.elasticsearch.pool_max_per_route: 8000
os: sudo sysctl -w net.core.rmem_default=131072000
