Why mutate will influence different pipelines?

Elastic Stack Logstash
1

Hi
I'd like to process the same input data in different ways to get different results, so I'm trying to do with multiple pipelines. However, I notice that using mutate will influence other pipelines. Does anyone know how to avoid this situation?

Thanks in advance!

Here is my pipelines.yml

    - pipeline.id: upstream
      config.path: "/usr/share/logstash/pipeline/upstream.conf"
    - pipeline.id: first-aggregator-downstream
      queue.type: persisted
      config.path: "/usr/share/logstash/pipeline/first-aggregator.conf"
    - pipeline.id: second-aggregator-downstream
      queue.type: persisted
      config.path: "/usr/share/logstash/pipeline/second-aggregator.conf"

Here is my upstream.conf

input {
      generator {
        lines => [
          "line1"
        ]
        count => 1
      }
    }
    
    output {
      pipeline {
        send_to => [
          firstAggregatorAddress,
          secondAggregatorAddress
        ]
      }
    }

Here is my first-aggregator.conf

input {
      pipeline {
        address => firstAggregatorAddress
      }
    }
    
    filter {
      mutate {
        add_field => { "first" => "first" }
      }
    }
    
    output {
      stdout { codec => json }
    }

Here is my second-aggregator.conf

input {
      pipeline {
        address => secondAggregatorAddress
      }
    }
    
    filter {
      mutate {
        add_field => { "second" => "second" }
      }
    }
    
    output {
      stdout { codec => json }
    }

The results are

  {
    "@version" => "1",
    "host" => {
    "name" => "hostname"
  },
    "second" => [
    [0] "second",
    [1] "second",
    [2] "second",
    [3] "second",
    [4] "second"
  ],
    "message" => "line1",
    "event" => {
    "original" => "line1",
    "sequence" => 0
  },
    "first" => [
    [0] "first",
    [1] "first",
    [2] "first",
    [3] "first",
    [4] "first"
  ],
    "@timestamp" => 2023-03-22T05:47:51.940537527Z
  }
2

Can anyone help?

3

Something is not right, when you use pipelines.yml the pipelines are independent from each other, so the input, filters and output cannot interfere with other pipelines.

How are you running logstash? What do you have in your logstash logs? Please share your logs.

Also, it is path.config, not config.path.

1 Like
4

Just replicate your pipeline and it is working fine, the output you shared is not what the pipelines you shared would generate, so it is probably related to how you are running logstash.

This is the output I got, and what you should've got since you are using the codec in the stdout output as json.

{"event":{"sequence":0,"original":"line1"},"@timestamp":"2023-03-31T03:10:34.298362520Z","@version":"1","message":"line1","host":{"name":"weiss"},"first":"first"}
{"event":{"sequence":0,"original":"line1"},"@timestamp":"2023-03-31T03:10:34.298362520Z","@version":"1","message":"line1","host":{"name":"weiss"},"second":"second"}
1 Like
5

Thank you for confirming that. I'll try it.

6

Oh, I see. But that's weird that it doesn't show any error config messages.

7

You probably are not using the pipelines.yml, but you didn't share how you are running Logstash or any logs.

8

I'm using k8s to run logstash and mount aforementioned config files to /usr/share/logstash/pipeline. What I can't understand is why there are added field values if the pipelines.yml is not used.

9

Thank you @leandrojmp . Thanks to you. I finally noticed the message:

{"level":"WARN","loggerName":"logstash.config.source.multilocal","timeMillis":1680243606102,"thread":"LogStash::Runner","logEvent":{"message":"Ignoring the 'pipelines.yml' file because modules or command line options are specified"}}

This is the cause of problem.

10

Finally I made it work by removing path.config from logstash.yml. Sorry for not providing this here.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.