In our logstash server 6 CPUs we have. Is this implies pipeline.worker value can be set to 6 ?
It implies it will default to 6 unless you set it to another value.
I don't know what is the default value of pipeline.worker i want to say does no of CPU cores is the value of pipeline.worker for example if 9 core cpu we have pipeline.worker value can be set between 1 to 9?
That is described here. Search for pipeline.workers on that page.
It defaults to the value of java.lang.Runtime.getRuntime.availableProcessors. Depending on what your pipeline does that might be too low, or it might be too high. If throughput is critical you would have to benchmark it for different values of pipeline.workers.
The default might be too high if you have two different CPU intensive pipelines in the same logstash instance. In that case you might want to reduce .workers to 4, or even 3 to reduce thread contention (this contention can greatly slow things down).
The default might be too low if your pipeline uses an http filter to call an expensive API, or a jdbc_streaming filter that runs an expensive SQL query. In this case the pipeline will spend time waiting for the remote server, which frees up the CPU for additional pipeline workers to use.
1 Like