Hello everyone,
I tried to set up a netflow analytic platform for my network but I think some data is missing.
I searched in my logs and haven't found any warnings or errors.
I use a Cent OS 6.8 Server with 32 GB Ram and 8 Cores.
My configurations looks like this:
On the same machine (I only have this server, splitting/creating more nodes atm. not possible):
1 logstash node
1 elasticsearch node
1 kibana node
"pipeline.workers"=>8,
"pipeline.batch.size"=>125,
"pipeline.batch.delay"=>5,
"pipeline.max_inflight"=>1000
max number of threads = 2048
vm.max_map_count = 262144
ES_JAVA_OPTS="-Xms10g -Xmx10g"
"number_of_shards": 1,
"number_of_replicas": 0
"index.refresh_interval": "60s"
1 Device vs 20 Devices:
Devices are Cisco Nexus 7k (NX OS).
When I sent Traffic from one Device to elasticsearch, my traffic data looks like this:
When I add 19 more devices, my traffic drops. Every other device should send as much traffic and packets as the single device:
Here some other comparisons (Left side is 20 devices / right side 1 device):
You can see that the index rate and the flows over time value do not change,
although there should be much more data.
Elastic vs dev/null:
When I try to send all logs to dev/null rather than to elasticsearch my event rate changes (left side to elastic / right side to dev/null):
Do you have any ideas, what could be my bottleneck?
The CPU is only at 25% and the ram isn't fully used too.
There aren't any warnings etc. and the cluster state is green.
Writing to the hard disk, shouldn't be the problem either.
If you have any ideas or if you need more informations,
please let me know.
i would really appreciate it, if you can help me.
thank you
Chris