# Pipelines not working but work when launched separately

**URL:** <https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330>\
**Category:** Logstash\
**Created:** [October 28, 2018, 9:43am UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330 "2018-10-28T09:43:13Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [October 28, 2018, 9:43am UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/1 "2018-10-28T09:43:13Z")

</div>

Has anyone managed to get pipelines working? Here are my observations:

1. My pipelines.yml file:

After launching logstash:  
`logstash --log.leve=debug -f .etc/logstash/mypipeline.yml`  
it fails with the following info:

:message=\> @Expected one of the #, input, filter, output at line 4

And that doesnt make any sense, as both of the config files mentioned in the pipeline above work perfectly when launched separately. So there is something wrong with the mypipeline.yml file.

1. concatenating two logstash files into one (echo ... \>\> combined.config) is readable by logstash, but second output also collects data from the first output and two indexes are being duplicated. So this approach doesn't work.

2. My next approach is to use conditionals to select desired output...which I don't really like as I'd like to use the power of pipelines.

Any ideas?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 28, 2018, 9:50am UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/2 "2018-10-28T09:50:33Z")

</div>

That is not now you pass pipeline definitions to Logstash. If you look at [the documentation](https://www.elastic.co/guide/en/logstash/6.4/multiple-pipelines.html) the pipeline definition should be placed in a file called `pipelines.yml` in the setting folder.

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [October 28, 2018, 1:15pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/3 "2018-10-28T13:15:35Z")

</div>

And that should solve the issue?  
Why can't I define other location for the pipeline.yml file?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 28, 2018, 1:22pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/4 "2018-10-28T13:22:35Z")

</div>

Does it not solve the issue? The files referenced in the `pipelines.yml` file can be in any location. If you want the `pipelines.yml` file in a different location you can specify it through the `path.settings` setting in the logstash.yml file.

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [October 28, 2018, 6:21pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/5 "2018-10-28T18:21:54Z")

</div>

Ok will check this in 2hrs. Meanwhile, could you please tell me what's the maximum number of input files for the logstash? I have seen this information somewhere (max of 2) and I think I run into same issue.  
When I use YML file with more than 2 inputs (I had 3), only third one has been processed.  
Is there a limit or some kind of common mistake or something I forgot about?

Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 28, 2018, 6:28pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/6 "2018-10-28T18:28:22Z")

</div>

I am not sure I understand your question. Could you perhaps give an example and show your configuration?

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [October 28, 2018, 6:33pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/7 "2018-10-28T18:33:35Z")

</div>

WORKS:

```auto
input {
  file {
    path => "/tmp/test_log.txt"
    type => "syslog"
  }
}

```

STILL WORKS:

```auto
input {
  file {
    path => "/tmp/test_log.txt"
    type => "syslog"
  }
}

input {
  file {
    path => "/tmp/test_json_log.txt"
  }
}

```

FAILS (and imports last input data only):

```auto
input {
  file {
    path => "/tmp/test_log.txt"
    type => "syslog"
  }
}

input {
  file {
    path => "/tmp/test_json_log.txt"
  }
}

input {
  file {
    path => "/tmp/final_input_log.txt"
  }
}

```

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [October 31, 2018, 1:51pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/8 "2018-10-31T13:51:27Z")

</div>

so it turns out it fails to fetch the input sources wht the number of iinputs is 3 lol 😉

- 1 input source - ok
- 2 input sources - ok
- 3 input sources - fails
- 4+ input sources - ok ... 😳

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 31, 2018, 1:59pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/9 "2018-10-31T13:59:49Z")

</div>

Is there potentially a problem with one of the input files?

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [November 1, 2018, 1:24pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/10 "2018-11-01T13:24:52Z")

</div>

nope, all work fine when launched separately.  
I am using git to check for the differences - nothing found, no 'magic' characters etc...

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 1, 2018, 3:12pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/11 "2018-11-01T15:12:40Z")

</div>

That is very, very odd.... Don't really have any suggestions or ideas.

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [November 16, 2018, 1:59pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/12 "2018-11-16T13:59:51Z")

</div>

I have upgraded logstash to the latest version and now it is ok.

Second question: How do I pass logstash.yml in the cli? I tried using --path.settings and using -f, but then path.settings [my\_pipelines.yml\_location] is being treated as config file and it throws an error that input/output was expected

---

<div class="post-metadata">

**Author:** ![wheelq](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wheelq/32/35893_2.png) [@wheelq](https://discuss.elastic.co/u/wheelq)\
**Post date:** [December 4, 2018, 7:43pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/13 "2018-12-04T19:43:24Z")

</div>

I was just thinking, how should I . run logstash so it picks up one of the pipelines? Currently I got:  
live\_pipelines.yml  
debug\_pipelines.yml  
test\_pipelines.yml

When I start logstash without any parameters, it looks for pipelines.yml.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 1, 2019, 7:43pm UTC](https://discuss.elastic.co/t/pipelines-not-working-but-work-when-launched-separately/154330/14 "2019-01-01T19:43:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.