Multipipeline Issues

Elastic Stack Logstash
1

ELK 6.1.1 stack running on Windows Server 2012R2
.\logstash -f path\pipelines.yml

pipelines.yml

  • pipeline.id: Beats
    path.config: "D:/ELKStack/LogStash/config/pipeline_beats.config"
  • pipeline.id: NMap
    path.config: "D:/ELKStack/LogStash/config/pipeline_nmap.config"
    queue.type: memory

I get the following error logged:

[2018-01-27T23:38:37,700][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after ", :backtrace=>["D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/pipeline.rb:171:in initialize'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:105:in block in execute'", "D:/ELKStack/LogStash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:ininterval'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/agent.rb:94:in execute'", "D:/ELKStack/LogStash/logstash-core/lib/logstash/runner.rb:343:inblock in execute'", "D:/ELKStack/LogStash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

If I copy and paste what is in my beats pipeline config file into the pipelines.yml, I get no errors and everything is fine. What is the error message pointing to that is wrong in my configs?

2

Either pipeline_beat.config or pipeline_nmap.config has a syntax error. By the looks of it the problem is at the very beginning. Perhaps you've edited the files with an editor that adds a byte-order mark? You can check this with a hex editor.

3

The pipelines.yml file should be placed in the settings folder, and can as far as I know not be passed through the -f flag. Place the file in the correct location and then try starting Logstash without the -f flag.

4

Gentlemen, thank you for your responses. The pipeline.yml is in the config folder, I should have made my example more clear. As for the editor, I used Notepad++, I'll look into the byte mark ordering. I could see the error was telling me it didn't like the first instruction it saw in the file, but the syntax makes it seem like it's not compatible with multi-pipelining.

5

Did some Googlin' and Notepad++ has two encoding settings, UTF-8 and UTF-8-BOM. All my config files are using UTF-8 encoding, so that doesn't appear to be the issue either. I also verified that UTF-8-BOM does not mean UTF-8 without BOM.

Any other ideas? Anything else about my setup you need to help me with the issue?

6

The path.settings directory where the pipelines.yml file should be located is the generally the same one as where the logstash.yml file is located. Is that where you have stored it?

7

I am not using a modified path.settings configuration, everything resides in the \logstash\config folder. I have tried running with the custom pipeline config files in logstash\config\pipelines folder and with them in the \logstash\config folder, both have the same behavior.

8

Bump

9

Hi @wwalker , I met the same issue.

we resolved by arranging the line ( I guess it is a kind of bug). It is related to windows environnement

change your line to :

path.config: D:\ELKStack\LogStash\config\pipeline_beats.config

in fact ,it seems that double quotes are our problem. also, keep your backslash as on windows.

KR

ld

10

I just discovered a way to make it work last night on an unrelated ElasticStack by omitting the drive letter, though I don't recall if I included double quotes or not.

path.config: /ELKStack/LogStash/config/pipeline_beats.config

11

path.config: /ELKStack/LogStash/config/pipeline_beats.config

there is a space ': /ELK' please remove it and try.

Please read below solution also for your reference.

12

I'm sorry....what?

13

in your file there is a space in below line

path.config: /ELKStack/LogStash/config/pipeline_beats.config ---space
path.config:/ELKStack/LogStash/config/pipeline_beats.config ---without space try this

14

Ah, I see what you're saying....which is odd because, as I said in the post you are quoting, by removing the drive letter, the issue was resolved. This thread is specifically about Windows installations, where full paths have drive letters so I could see where you could be confused if you thought I was talking about Linux.

15

yes i was talking about linux. bcoz earlier we faced same issue and got resolved by this.

and also provide this solution to others and it works so i thought u might have same issue.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.