Netflow -> ELK, poor performance


our setup (pilot): 10 neflow exporters (cca 10k events/sec) to logstash with netflow modul to elastic. In the beginning (after restart logstash) everything is ok but after 10 min input is droping to 1k events/sec (kibana netflow discover attach). We were playing with this parameters to even get 10k per sec in start (bellow). Please help/advice.

ELK: 1 node vmware redhat 7.6, 36cores, 128GB ram, jvm 32 GB
pipeline.batch.size: 3000
pipeline.batch.delay: 5
pipeline.workers: 96


  • name: netflow
    var.input.udp.port: 2056
    var.input.udp.buffer_size: 655360
    var.input.udp.queue_size: 40000
    var.input.udp.workers: 36
    var.input.udp.receive_buffer_bytes: 134217728
    var.elasticsearch.ssl.enabled: false
    var.kibana.scheme: http
    var.kibana.ssl.enabled: false
    var.kibana.ssl.verification_mode: disable
    var.output.elasticsearch.pool_max: 80000
    var.output.elasticsearch.pool_max_per_route: 8000

os: sudo sysctl -w net.core.rmem_default=131072000

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.