Pipeline to Pipeline Output Variable Expansion

Elastic Stack Logstash
1

Hello,

I'm trying to organize/refactor our Logstash (6.5.4) pipelines by using the Pipeline-to-Pipeline beta feature. We ingest a lot of custom log formats in addition to logs from the Filebeat (6.5.4) modules, so I have setup a forked-distributor pattern where our custom logs get sent to their own pipelines and the modules get sent to theirs (I've converted the Elasticsearch ingest pipelines for use by Logstash).

Instead of writing each one out by hand (e.g. if [fileset][module] == "mysql" etc), I had hoped to do this more dynamically like so:

if ([fileset][module] and [fileset][name]) {
     pipeline { send_to => "%{[fileset][module]}-%{[fileset][name]}" }
}
else if ([fileset][module]) {
     pipeline { send_to => "%{[fileset][module]}-log"}
}
else if customlogs... etc

This sort of variable expansion works well enough for Elasticsearch outputs, e.g. index, but Logstash treats it as a string.

Attempted to send event to '%{[fileset][module]}-%{[fileset][name]}' but that address was unavailable.

I'm not sure if this is a bug, or more likely, if I've overlooked something that would suggest this method of converting ingest pipelines to Logstash pipelines and using them in this manner is ill advised, but I would appreciate any feedback regardless.

Thanks,
James S.

4 Likes
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.