Logstash input speed slow, around 8,000events / sec writing to local file

Elastic Stack Logstash
1

Hello guys, thanks for noticing this question.
I am trying to build up a centralized logging system using ELK framework, but the speed is very slow. I checked and the problem is on logstash. I used a generator to generate logs as input, and output to a local file. And I am achieving a throughput of 8,000/sec. Is this normal? Because I am expecting a much faster speed. Here is my .conf file:

input {
generator {
type => foo
}
}

output {
file{
path => "/home/checkIII"
}
}

I tried to modify the flush_interval or the -w, but nothing works. I would be really appreciated if you guys can give me some advice.

Come on guys...I just need someone to tell me is this normal or not. Desperately needing help here.

1 Like
2

Have you experimented with increasing the number of threads for the generator plugin?

3

Hi Christian, thanks for replying. Yes, I have tried to increase the threads of the generator plugin, but the throughput remains the same, still 8,000/sec. I guess that is not the bottle neck. And I wonder if there are some other logstash configurations that I need to adjust, because no matter I read from a file or input from a generator, the throughput does not change. Besides, neigher of the cpu utilization and memory utilization is high, the disc i/o is far lower than the threshold.

4

I ran your configuration with default number of threads and workers on Logstash 1.4.2 on my MacBook Air and got a bit over 52k events generated per second. What kind of hardware are you running the test on?

5

Well this is weird...
I ran it on an AWS m1.xlarge machine. It hase 4 core cpu and 16G ram, mounted a 1.5TB ssd. It should have a much better performance than a macbook air...

6

18k with that config on a thinkpad i7 SSD 500g

7

Same issue here, and I have a little more info.

In our scenario, log stash pulls from kafka, processes the docs, and writes them to null {} at 17k eps. The limit here is mostly due to our parsing and this keeps all 12 cpus assigned busy at near 100%. This is working great.

However, if we send the output to elastic search, playing with workers, flush_size, etc, we get almost 9k eps. Elasticsearch is not busy.

What we found just now is that switching the output to file - gets the same results as using elastic search - only 8300 eps. The disk is hardly being used, and CPUs are no where near busy. (iostat shows less than 1% disk utilization.)

If we switch back to null {} output, it goes to > 17k eps, and the CPUs are all busy. So, something appears wrong with how the output plugins pull from the filter queue. They seem not to be able to pull at over 8 to 9k eps for some reason. Both file and elastic search output plugins have the same limit inside Logstash.

The disk we are using, based on copying a file with JSON docs in it, can copy these docs at over 109k eps. Iostat shows this at almost 100% utilization. The disk I/O is not the problem.

It would be great if output plugins could pull from the filter section faster than 8 to 9k eps. In the ELK system we are building, this limitation means we have to use a lot more Logstash systems that we should.

2 Likes
8

Does changing the number of workers in the Elasticsearch output plugin make any difference?

9

Hey all, thought I'd follow up.

We got past most of our Logstash issues. We pull from kafka, and can pull fast, maybe 100k eps. We run it through our filters and geoip enrichment and, with -w 8 or higher, we can get over 13k eps parsed and ready for output.

The last step is to output to Elasticsearch, and we can get 10 to 13k eps into Elasticsearch cluster per LS box.

The main settings are the -w setting which controls how many threads run the filters. On Linux, you can use "top" with the "H" command in top, to see the threads. LS labels them kafka, filter and elasticsearch to make it easy to see what is going on. If the filter threads are not running at 100% CPU, then they aren't the problem.

The other main setting is memory. For us, we needed to increase the heap space for Logstash to 4G. 2G made a big improvement, but 4G was even faster. Over 4G was not better so we stopped at 4G memory.

In sum, we use 8 to 10 cores and 4G memory setting to achieve up to 13k eps into our cluster. Since LS 2.0, it chooses the number of cores better. It used to have a default of one, now its 1/2 the CPUs on the box, I believe, which works great for us. The memory for LS is set in /etc/sysconfig/logstash.

I hope this helps.

10

I'm having similar issue with my kafka input config...how did you configure your input settings.....did you configure any settings like "fetch_min_bytes"?

whats weird is that I have 16 partitions so I'm running 14 consumer threads and still, I'm only able to get about 3k/sec. I also have -w 40 but that still didn't help..