Me and my team currently having an issue with netflow data ingestion using logstash / filebeat. The data seems to be massively dropped to around 1/7 (using logstash) and 1/100 (using filebeat) when we compare to the actual data. The dropped data also stated as below picture on the filebeat log.
We have already tried:
-Tuning the JVM heap, workers, and bulk size
-Upgrading Elasticsearch cluster from single node to 3 nodes (each on different server)
-Increasing the index primary shard to 3
-Temporarily removing all logstash processes (logstash filter section)
-Add the fourth server to seperate the logstash/filebeat service (used to run on the same server as Elasticsearch)
But still no significant changes on the data after those tweaking.
For additional info, we also have tried to ingest a netflow data from a different device which the data rate is way smaller than this one and it actually works with the same cluster condition (around 99% similar with the actual data).
Any advices as how to resolve this issue and/or best practices would be much appreciated.
Thank you.