Number of batch workers

Elastic Stack Logstash
1

I have a platform where filebeat , logstash , elasticsearch , kibana is all installed in one server.

In the document there is a sentence explaining about the number of workers.

The number of workers may be set higher than the number of CPU cores since outputs often spend idle time in I/O wait conditions.

I am not expertise with application programming so could someone elaborate why you need to increase number of workers if output spend idle time in I/O wait conditions?

I am concerned about this since I have 8 core (HT enabled?) environment and where number of workers are 8 (default).

[root@IntegrationTest-Env creationline]# lscpu
    Architecture:          x86_64
    CPU op-mode(s):        32-bit, 64-bit
    Byte Order:            Little Endian
    CPU(s):                8
    On-line CPU(s) list:   0-7
    Thread(s) per core:    1
    Core(s) per socket:    8
    Socket(s):             1
    NUMA node(s):          1
    Vendor ID:             GenuineIntel
    CPU family:            6
    Model:                 63
    Model name:            Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz
    Stepping:              2
    CPU MHz:               2394.436
    BogoMIPS:              4788.87
    Hypervisor vendor:     Microsoft
    Virtualization type:   full
    L1d cache:             32K
    L1i cache:             32K
    L2 cache:              256K
    L3 cache:              30720K
    NUMA node0 CPU(s):     0-7

    [root@IntegrationTest-Env creationline]# curl -XGET localhost:9600/_node/pipeline?pretty
    {
      "host" : "Honda-IntegrationTest-Env",
      "version" : "5.0.0-alpha5",
      "http_address" : "127.0.0.1:9600",
      "pipeline" : {
        "workers" : 8,
        "batch_size" : 125,
        "batch_delay" : 5
      }
    }[root@IntegrationTest-Env creationline]# cat /etc/redhat-release
2

I am not expertise with application programming so could someone elaborate why you need to increase number of workers if output spend idle time in I/O wait conditions?

You obviously don't have to increase the number of pipeline workers, but if threads are sleeping waiting for I/O then they're not loading the CPU, meaning you have room to increase the concurrency by raising the number of workers without overloading the system. In your case you should probably be careful and maybe reduce the number of workers since the machine isn't exclusively used for Logstash.