# Many workers whereas only 1 expected

**URL:** <https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942>\
**Category:** Logstash\
**Created:** [January 3, 2018, 2:46pm UTC](https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942 "2018-01-03T14:46:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nicolas\_Guyomard](https://avatars.discourse-cdn.com/v4/letter/n/258eb7/32.png) [@Nicolas\_Guyomard](https://discuss.elastic.co/u/Nicolas_Guyomard)\
**Post date:** [January 3, 2018, 2:46pm UTC](https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942/1 "2018-01-03T14:46:52Z")

</div>

I use aggregate filter plugin to extract metrics from the log of my application using the following config:

```
input {
   file {
      path => "application.log"
      start_position => "beginning"
     sincedb_path => "/dev/null"
   }
}

filter {
    grok {
        match => {
            "message" => "%{LOG_PATTERN}"
        }
    }

    if ([module] == "input") {
        aggregate {
            task_id => "%{req_id}"
            code => "
                map['in_timestamp'] = event.get('timestamp')
                map['module_time'] = {}
            "
            map_action => "create"
        }
    }
    else if ([module] == "output") {
        aggregate {
            task_id => "%{req_id}"
            code => "
                event.set('module_time', map['module_time'])
                event.set('in_timestamp', map['in_timestamp'])
            "
            map_action => "update"
            end_of_task => true
        }
    }
    else {
        aggregate {
                task_id => "%{req_id}"
                code => "
                    map['module_time'][event.get('module')] = event.get('timestamp')
                "
                map_action => "update"
        }
    }
}

output {
   file {
      path => "application.json"
      codec => "rubydebug"
   }
}

```

I set Logstash filter workers to 1 both in logstash.yml (pipeline.workers: 1) and command line (-w 1) but it looks like many workers are used.  
I see in the output that the timestamp of the event corresponding to line 2016 of my log is before the timestamp of the event corresponding to line 2002 of my log.  
As both lines correspond to the same req\_id but line 2016 corresponds to output module, I am loosing data.

I see that I can fix the issue for this req\_id by changing the value of "pipeline.batch.size" in logstash.yml, but the issue occurs for other req\_id.

What am I doing wrong?

How can I check the number of workers used by Logstash ?

---

<div class="post-metadata">

**Author:** ![Nicolas\_Guyomard](https://avatars.discourse-cdn.com/v4/letter/n/258eb7/32.png) [@Nicolas\_Guyomard](https://discuss.elastic.co/u/Nicolas_Guyomard)\
**Post date:** [January 4, 2018, 1:45pm UTC](https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942/2 "2018-01-04T13:45:57Z")

</div>

The node info API (curl -XGET 'localhost:9600/\_node/pipelines?pretty') inform me that there is only 1 worker:

```
  "pipelines" : {
    "main" : {
      "workers" : 1,
      "batch_size" : 1000,
      "batch_delay" : 10,
      "config_reload_automatic" : false,
      "config_reload_interval" : 3000000000,
      "dead_letter_queue_enabled" : false
    }

```

So how to explain that the events are interlaced ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 1, 2018, 1:45pm UTC](https://discuss.elastic.co/t/many-workers-whereas-only-1-expected/113942/3 "2018-02-01T13:45:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.