Setup I have multiple machines running a containerized Logstash 7.6. These machines have 32 core and 256GB RAM. I am using these nodes to process network flow data based on Elastiflow . These nodes are listening for data using the Sflow codec and pushing data to an ES cluster of 32 nodes, each with t…

Hi I am not familiar with sflow but I am curious. How many events per second and how does the cpu usage look like? Regards Kim

Great question! I'll use one node as an example: The busy instances are receiving ~17k packets per second. [image] CPU usage is relatively low: [image] Yet I'm seeing many UDP receive buffer errors: [image]

interesting .. Have you read this topic : https://discuss.elastic.co/t/netflow-codec-udp-receive-errors/127111/7

That is a very interesting read. I agree that I'd likely benefit from breaking my large, monolithic Logstash instances into smaller containers with a load balancer, however I can probably only realistically run 4 instances per physical host before seeing performance degradation. I suppose that may …

We have one use case where we have one pipeline to receive syslog from our fw. This is high load 20.000+ eps. We just output to Kafka in that pipeline. You can then have any number of additional logstash nodes pulling data from. Kafka and parse events and ingest to elastic. This architecture has wor…

Ah interesting - so your message queue is basically your load balancer in this case.. That might work for the load balancing solution that I clearly need. It looks like I need to apply basically three fixes to get the desired output: Horizontal scaling of Logstash instances i. Question remains: s…

I have two things for you to try... Go to the 4.x-dev branch of ElastiFlow and get the file sysctl.d/87-elastiflow.conf. Put this in /etc/sysctl.d and restart the server. This increases kernel buffers that will help with receiving a high volume of UDP packets. (NOTE: you could change these settin…

You're the man Rob. I'll try that today and let you know the results.

Thanks for the recommendations again. I received about a 10-12% gain in throughput by updating the nice level for Logstash. The sysctl optimizations provided about another 5% of gain. I've horizontally scaled my Logstash instances in my container using Nginx as a load balancer as seen here: [imag…

If Kafka is in the mix, I would split the ElastiFlow pipeline into two parts. A collector/decoder and a post-processor with Kafka in the middle. On my to-do list is a video where I explain why this is a good idea for any high volume UDP use-cases. You will also want to use the recommendations for k…

Increasing Logstash Throughput When the Codec is the Bottleneck

Elastic Stack Logstash

seth.yes (Seth S) March 18, 2020, 6:21pm 3

Great question!

I'll use one node as an example:

The busy instances are receiving ~17k packets per second.

CPU usage is relatively low:

Yet I'm seeing many UDP receive buffer errors:

Topic		Replies	Views
Netflow Codec : UDP receive errors Logstash	12	3462	May 11, 2018
Only get 900 netflow events/sec from 4 vCPU LS node Logstash	3	2491	May 26, 2018
Improving capacity to handle high netflow rates Logstash	3	631	November 15, 2017
Netflow High CPU utilization and poor throughput Logstash	4	2062	August 30, 2017
UDP/Netflow Performance Logstash	4	2603	July 6, 2017

Increasing Logstash Throughput When the Codec is the Bottleneck

Related topics