Multipipeline Issues

ELK 6.1.1 stack running on Windows Server 2012R2
.\logstash -f path\pipelines.yml

pipelines.yml

  • pipeline.id: Beats
    path.config: "D:/ELKStack/LogStash/config/pipeline_beats.config"
  • pipeline.id: NMap
    path.config: "D:/ELKStack/LogStash/config/pipeline_nmap.config"
    queue.type: memory

I get the following error logged:

[2018-01-27T23:38:37,700][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ", :backtrace=>["D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/pipeline.rb:171:in initialize'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "D:/ELKStack/LogStash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:94:in execute'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/runner.rb:343:inblock in execute'", "D:/ELKStack/LogStash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

If I copy and paste what is in my beats pipeline config file into the pipelines.yml, I get no errors and everything is fine. What is the error message pointing to that is wrong in my configs?

Either pipeline_beat.config or pipeline_nmap.config has a syntax error. By the looks of it the problem is at the very beginning. Perhaps you've edited the files with an editor that adds a byte-order mark? You can check this with a hex editor.

The pipelines.yml file should be placed in the settings folder, and can as far as I know not be passed through the -f flag. Place the file in the correct location and then try starting Logstash without the -f flag.

Gentlemen, thank you for your responses. The pipeline.yml is in the config folder, I should have made my example more clear. As for the editor, I used Notepad++, I'll look into the byte mark ordering. I could see the error was telling me it didn't like the first instruction it saw in the file, but the syntax makes it seem like it's not compatible with multi-pipelining.

Did some Googlin' and Notepad++ has two encoding settings, UTF-8 and UTF-8-BOM. All my config files are using UTF-8 encoding, so that doesn't appear to be the issue either. I also verified that UTF-8-BOM does not mean UTF-8 without BOM.

Any other ideas? Anything else about my setup you need to help me with the issue?

The path.settings directory where the pipelines.yml file should be located is the generally the same one as where the logstash.yml file is located. Is that where you have stored it?

I am not using a modified path.settings configuration, everything resides in the \logstash\config folder. I have tried running with the custom pipeline config files in logstash\config\pipelines folder and with them in the \logstash\config folder, both have the same behavior.

Bump

Hi @wwalker , I met the same issue.

we resolved by arranging the line ( I guess it is a kind of bug). It is related to windows environnement

change your line to :

path.config: D:\ELKStack\LogStash\config\pipeline_beats.config

in fact ,it seems that double quotes are our problem. also, keep your backslash as on windows.

KR

ld

I just discovered a way to make it work last night on an unrelated ElasticStack by omitting the drive letter, though I don't recall if I included double quotes or not.

path.config: /ELKStack/LogStash/config/pipeline_beats.config

path.config: /ELKStack/LogStash/config/pipeline_beats.config

there is a space ': /ELK' please remove it and try.

Please read below solution also for your reference.

I'm sorry....what?

in your file there is a space in below line

path.config: /ELKStack/LogStash/config/pipeline_beats.config ---space
path.config:/ELKStack/LogStash/config/pipeline_beats.config ---without space try this

Ah, I see what you're saying....which is odd because, as I said in the post you are quoting, by removing the drive letter, the issue was resolved. This thread is specifically about Windows installations, where full paths have drive letters so I could see where you could be confused if you thought I was talking about Linux.

yes i was talking about linux. bcoz earlier we faced same issue and got resolved by this.

and also provide this solution to others and it works so i thought u might have same issue.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.