Pipelines not working but work when launched separately

Elastic Stack Logstash
1

Has anyone managed to get pipelines working? Here are my observations:

  1. My pipelines.yml file:

    • pipeline.id: httppipeline
      path.config: "http.conf"
    • pipeline.id: ftppipeline
      path.config: "ftp.conf"

After launching logstash:
logstash --log.leve=debug -f .etc/logstash/mypipeline.yml
it fails with the following info:

:message=> @Expected one of the #, input, filter, output at line 4

And that doesnt make any sense, as both of the config files mentioned in the pipeline above work perfectly when launched separately. So there is something wrong with the mypipeline.yml file.

  1. concatenating two logstash files into one (echo ... >> combined.config) is readable by logstash, but second output also collects data from the first output and two indexes are being duplicated. So this approach doesn't work.

  2. My next approach is to use conditionals to select desired output...which I don't really like as I'd like to use the power of pipelines.

Any ideas?

2

That is not now you pass pipeline definitions to Logstash. If you look at the documentation the pipeline definition should be placed in a file called pipelines.yml in the setting folder.

3

And that should solve the issue?
Why can't I define other location for the pipeline.yml file?

4

Does it not solve the issue? The files referenced in the pipelines.yml file can be in any location. If you want the pipelines.yml file in a different location you can specify it through the path.settings setting in the logstash.yml file.

5

Ok will check this in 2hrs. Meanwhile, could you please tell me what's the maximum number of input files for the logstash? I have seen this information somewhere (max of 2) and I think I run into same issue.
When I use YML file with more than 2 inputs (I had 3), only third one has been processed.
Is there a limit or some kind of common mistake or something I forgot about?

Thanks

6

I am not sure I understand your question. Could you perhaps give an example and show your configuration?

7

WORKS:

input {
  file {
    path => "/tmp/test_log.txt"
    type => "syslog"
  }
}

STILL WORKS:

input {
  file {
    path => "/tmp/test_log.txt"
    type => "syslog"
  }
}

input {
  file {
    path => "/tmp/test_json_log.txt"
  }
}

FAILS (and imports last input data only):

input {
  file {
    path => "/tmp/test_log.txt"
    type => "syslog"
  }
}

input {
  file {
    path => "/tmp/test_json_log.txt"
  }
}

input {
  file {
    path => "/tmp/final_input_log.txt"
  }
}
8

so it turns out it fails to fetch the input sources wht the number of iinputs is 3 lol :wink:

  • 1 input source - ok
  • 2 input sources - ok
  • 3 input sources - fails
  • 4+ input sources - ok ... :flushed:
9

Is there potentially a problem with one of the input files?

10

nope, all work fine when launched separately.
I am using git to check for the differences - nothing found, no 'magic' characters etc...

11

That is very, very odd.... Don't really have any suggestions or ideas.

12

I have upgraded logstash to the latest version and now it is ok.

Second question: How do I pass logstash.yml in the cli? I tried using --path.settings and using -f, but then path.settings [my_pipelines.yml_location] is being treated as config file and it throws an error that input/output was expected

13

I was just thinking, how should I . run logstash so it picks up one of the pipelines? Currently I got:
live_pipelines.yml
debug_pipelines.yml
test_pipelines.yml

When I start logstash without any parameters, it looks for pipelines.yml.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.