Pipeline to pipeline communication - logstash data

Elastic Stack Logstash
1

Pipeline1 :- ds_test.conf

input { generator { count => 1 lines => [ '' ] } }

filter {
    mutate { add_field => { "[eventHeader][applicationId]" => "A" "[eventHeader][objectCode]" => 1 } }
    mutate { add_field => { "[@metadata][ObjectCode]" => "%{[eventHeader][applicationId]}_%{[eventHeader][objectCode]}"} }

   #prune { whitelist_names => [ "itemId","IndexedOn","field","text","softDelete","normalizedText","type","appId","nounModifier","contactcode"] }
}

output { pipeline { send_to => [ds_test1] } }
=========================================
pipeline 2:- ds_test1.conf
input {
    pipeline { address => "ds_test1"}
}
filter {
     mutate { add_field => { "test" => "%{[@metadata][ObjectCode]}"} }
}

output{
    stdout { codec => rubydebug { metadata => true} }
}
=======================================================

am not able to see any debug output

command debug:- /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/ds_test.conf -f /etc/logstash/conf.d/ds_test1.conf --path.settings=/etc/logstash


please let me know how i can see  event and metadata in pipeline 2
2

you can’t use multiple -f in cli. only the last one will be used.

from the docs :

-f, --path.config CONFIG_PATH

Load the Logstash config from a specific file or directory. If a directory is given, all files in that directory will be concatenated in lexicographical order and then parsed as a single config file. Specifying this flag multiple times is not supported. If you specify this flag multiple times, Logstash uses the last occurrence (for example, -f foo -f bar is the same as -f bar ).

3

To run multiple pipelines you have to configure pipelines.yml, which is typically in /etc/logstash.

- pipeline.id: pipe1
  path.config: "/path/to/ds_test.conf"
- pipeline.id: pipe2
  path.config: "/path/to/ds_test1.conf"

Do not use -f since that will cause pipelines.yml to be ignored.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.