How far to push the CPU on logstash

[pk.241011]

Hi,

I have a 8 Core CPU with 16GB of RAM.
8 GB has been allocated to Logstash. Nothing else is running on the machine.

The logstash is running a single pipeline only.

I have seen that even when I configure the number of worker threads to 32, the max CPU usage is going only to 44%.

Can I keep experimenting and increasing the number of workers so that the CPU usage goes to somewhere around 75%?

Logstash is not doing very heavy data transformation.

Also there is no linear increase in the throughput. The throughput is increasing with increase in the workers but not that it is becoming double.

Any ideas?

[Badger]

@pk.241011

there is no linear increase in the throughput

Java performance will not scale directly with the number of threads. For example, for most JVMs some aspects of garbage collection are single-threaded, so only one CPU will be used during GC pauses. Various plugins use mutexes to protect shared resources that can only be accessed in a thread-safe manner by one thread at a time.

It is also possible for logstash to be limited by the throughput of the inputs it is reading from or the outputs it is writing to.

To investigate the effect of increasing the number of worker threads properly you would need to start collecting thread dumps and see what the additional threads are doing. If they are doing work then throughput will probably continue scaling. If they are sitting on mutexes then no additional work will get done (but CPU may increase -- in this case increased CPU usage is bad, not good).

Try to maximize throughput (events per minute), not CPU usage.

It is a very general Java performance analysis problem, not really a logstash question.