Hi,
If I set pipeline.workers > 1 then event dependent configuration, "%{}", appears to be disabled. Here is my test case.
#!/bin/bash
# This works
export TIX_VERSION=1
gunzip --stdout tmp/cart.gz | logstash --pipeline.workers 1 --path.config=logstash/cart_logstash.conf
# This does not
export TIX_VERSION=2
gunzip --stdout tmp/cart.gz | logstash --pipeline.workers 4 --path.config=logstash/cart_logstash.conf
The single worker creates a series of indices, with the document_id's set with document_id. The parallel worker creates one index, with %{} in its name, with one document whose id is %{id} and lots of deleted documents.
I'm running logstash 6.2.4, which brew tells me is current.
My config file is as follows:
input {
stdin {}
}
filter {
mutate {
gsub => ["message", "\"", "'"]
# Double quote problem https://discuss.elastic.co/t/csv-filter-quote-character-causing-csvparsefailure/121273/3 '
}
csv {
autodetect_column_names => "true"
remove_field => ["message"] # message is the unparsed row
separator => " " # Yes, that is supposed to be a literal tab
skip_empty_columns => "true"
skip_header =>"true"
}
date {
match => ["arrival_date", "YYYY-MM-dd HH:mm:ss"]
target => "arrival_date"
timezone => "UTC"
}
mutate {
remove_field => ["host"]
}
}
output {
elasticsearch {
document_id => "%{id}"
hosts => ["${ES_HOST}:${ES_PORT}"]
http_compression => "true"
index => "cart-%{slug}-v${TIX_VERSION}"
manage_template => "false"
password => "${ES_PASS}"
ssl => "true"
user => "${ES_USER}"
}
}
Have I missed something, or should I open an issue?