Multiple pipelines output to one cluster - only one of them picks up logs

[Ilya_Geller]

Hi,
I have a logstash node that has two pipelines that pick up different inputs and do separate things but both of them output to the same ES cluster into different indeces.
Both pipelines have the following defined:

    output{
        elasticsearch{
        ...
        }
    }

For some reason, when both pipelines are enabled, Logstash starts without any errors, and everything looks fine, but only one of the pipelines actually sends data into ES.
When I disable one pipeline, and only one of them is active at a time, each of them works fine.

Is there any known issue with more than one elasticsearch output configured when its the same cluster?

When I watch the logstash startup logs I can see the elasticsearch output initializing twice saying

"New Elasticsearch output..."
...
"New Elasticsearch output..."

Does this mean that the second one overrides the first one and so the first pipeline remains without an output?

[leandrojmp (Leandro Pereira)]

Can you share your pipelines.yml file and your pipelines configuration?

Having multiple elasticsearch outputs is pretty normal, this should not be the problem, but without more information about your pipelines is not possible to know what could be the problem.

[Ilya_Geller]

Thanks for the reply.

My pipelines.yml file is the default one, I haven't touched it.

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

As for my pipelines, I can't really share them since they are on a private network, however the general idea is:
pipeline 1:

input {
    file{
        ...pick up local files...    
    }
}

filter{
    multiple filters{
     ....
    }
}
output{
    elasticsearch{
        hosts => ["es_node"]    
        index => "index-1"
    }
}

pipeline 2:

input {
    http{
        ...get logs via http...
    }
}

filter{
    different filters{
     ....
    }
}
output{
    elasticsearch{
        hosts => ["es_node"]    
        index => "index-2"
    }
}

Logstash output is normal without errors

...
[main] Pipeline started {"pipeline.id"=>"main"}
...
 Successfully started Logstash API endpoint {:port=>9600}

I hope this is enough information...

[leandrojmp (Leandro Pereira)]

This config in pipelines.yml tells logstash that you have only one pipeline named main, and this pipeline is the content of all the *.conf files in the directory /etc/logstash/conf.d.

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

When logstash starts it will concatenate your files and group the inputs, filters and outputs, you will have something like this.

input {
    file { from your pipeline 1 file }
    http { from your pipeline 2 file }
}
filter {
    filters from pipeline 1
    filters from pipeline 2
}
output {
    elasticsearch { 
        hosts => ["es_node"]
        index => "index-1"
    }
    elasticsearch { 
        hosts => ["es_node"]
        index => "index-2"
    }
}

This works fine, but every event from the file input and from the http input will pass through every filter from both the pipelines files and should be stored in both index, since you do not have conditionals in your output block.

If this is not the behaviour you want, I would suggest that you change your pipelines.yml and use two different pipelines.

- pipeline.id: pipeline-1
  path.config: "/etc/logstash/conf.d/pipeline-1.conf"
- pipeline.id: pipeline-2
  path.config: "/etc/logstash/conf.d/pipeline-2.conf"

With this config your pipelines will be independent from each other.

Even when concatenating the files, I do not see any reason for only one input work at a time, but without seeing the filters and what your pipeline is really doing I can't tell much.

I would suggest that you split your pipelines using the above config for the pipelines.yml to try to find where the error could be.

[Ilya_Geller]

@leandrojmp

This config in pipelines.yml tells logstash that you have only one pipeline named main , and this pipeline is the content of all the *.conf files in the directory /etc/logstash/conf.d .

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"

When logstash starts it will concatenate your files and group the inputs , filters and outputs

Wow, I really didn't think about it that way. I thought that each .conf file in my conf.d directory is a separate pipeline, and thereforepipelines.yml just runs them all together, but I didn't figure it just makes all of them into one big pipeline.

I will give it a shot and try separating the two pipelines like you suggested and I hope it will do.

Thank you!