# Event dependent configuration broken with multiple pipelines?

**URL:** <https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869>\
**Category:** Logstash\
**Created:** [July 27, 2018, 4:21am UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869 "2018-07-27T04:21:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Steven\_Ensslen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steven_ensslen/32/26984_2.png) [@Steven\_Ensslen](https://discuss.elastic.co/u/Steven_Ensslen)\
**Post date:** [July 27, 2018, 4:21am UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869/1 "2018-07-27T04:21:50Z")

</div>

Hi,

If I set [pipeline.workers](https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html) \> 1 then [event dependent configuration](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html), "%{}", appears to be disabled. Here is my test case.

```
#!/bin/bash

# This works
export TIX_VERSION=1
gunzip --stdout tmp/cart.gz | logstash --pipeline.workers 1 --path.config=logstash/cart_logstash.conf

# This does not
export TIX_VERSION=2
gunzip --stdout tmp/cart.gz | logstash --pipeline.workers 4 --path.config=logstash/cart_logstash.conf

```

The single worker creates a series of indices, with the document\_id's set with `document_id`. The parallel worker creates one index, with `%{}` in its name, with one document whose id is `%{id}` and lots of deleted documents.

I'm running logstash 6.2.4, which `brew` tells me is current.

My config file is as follows:

```
input {
    stdin {}
}
filter {
    mutate {
        gsub => ["message", "\"", "'"]
        # Double quote problem https://discuss.elastic.co/t/csv-filter-quote-character-causing-csvparsefailure/121273/3 '
    }
    csv {
        autodetect_column_names => "true"
        remove_field => ["message"] # message is the unparsed row
        separator => "	" # Yes, that is supposed to be a literal tab
        skip_empty_columns => "true"
        skip_header =>"true"
    }
    date {
      match => ["arrival_date", "YYYY-MM-dd HH:mm:ss"]
      target => "arrival_date"
      timezone => "UTC"
    }
    mutate {
     remove_field => ["host"]
    }
}
output {
      elasticsearch {
        document_id => "%{id}"
        hosts => ["${ES_HOST}:${ES_PORT}"]
        http_compression => "true"
        index => "cart-%{slug}-v${TIX_VERSION}"
        manage_template => "false"
        password => "${ES_PASS}"
        ssl => "true"
        user => "${ES_USER}"
    }
}  

```

Have I missed something, or should I open an issue?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 27, 2018, 6:00pm UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869/2 "2018-07-27T18:00:21Z")

</div>

> If I set pipeline.workers \> 1 then event dependent configuration, "%{}", appears to be disabled.

That sounds extremely unlikely, but I don't really have a better suggestion to offer. What happens if you replace your elasticsearch output with `stdout { codec => rubydebug }`? Do all resulting events contain the required `id` and `slug` fields?

---

<div class="post-metadata">

**Author:** ![Steven\_Ensslen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steven_ensslen/32/26984_2.png) [@Steven\_Ensslen](https://discuss.elastic.co/u/Steven_Ensslen)\
**Post date:** [July 30, 2018, 8:43pm UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869/3 "2018-07-30T20:43:16Z")

</div>

`stdout { codec => rubydebug }` is giving me nonsense, like this:

> {  
> "@version" =\> "1",  
> "locationA" =\> "locationA",  
> "@timestamp" =\> 2018-07-30T20:35:38.032Z,  
> "0.00" =\> "0.00",  
> "2016-03-19 22:00:00" =\> "2015-11-11 21:00:00",  
> "0000788c-87da-4219-9379-dd7f0febd1ac" =\> "0001c971-cc81-4cf1-8ae5-843958777a2d",  
> "8" =\> "3",  
> "[Amanda.Example@notreal.com](mailto:Amanda.Example@notreal.com)" =\> "[Barry@example.com](mailto:Barry@example.com)",  
> "Amanda Example" =\> "Barry Fake Person",  
> "Venue A" =\> "Venue A"  
> }

It perhaps has failed to `autodetect_column_names`?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 1, 2018, 6:08am UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869/4 "2018-08-01T06:08:58Z")

</div>

Yes, that's most likely the case. I've never trusted the autodetect feature.

---

<div class="post-metadata">

**Author:** ![Steven\_Ensslen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steven_ensslen/32/26984_2.png) [@Steven\_Ensslen](https://discuss.elastic.co/u/Steven_Ensslen)\
**Post date:** [August 2, 2018, 3:32am UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869/5 "2018-08-02T03:32:28Z")

</div>

There is already [an issue for this in logstash-csv-filter](https://github.com/logstash-plugins/logstash-filter-csv/issues/65).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2018, 3:35am UTC](https://discuss.elastic.co/t/event-dependent-configuration-broken-with-multiple-pipelines/141869/6 "2018-08-30T03:35:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.