Logstash on Windows - Multiple Pipeline Problem

Is it possible to run multiple pipelines on the Windows version of Logstash? I can’t seem to get this to work for the life of me.

I have installed Logstash on Windows, and placed a pipelines.yml file in C:\Program Files\Logstash\config.

Here are the contents of the pipelines.yml

  • pipeline.id: pipelinedmarcxml
    path.config: "c:\Program Files\Logstash\config\pipelines\dmarcxml.d"
    pipeline.workers: 3

When I launch logstash.bat from C:\Program Files\Logstash\bin, I see this message logged:

• [ WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

Then at the end of the startup process the pipeline terminates – assuming because it can’t load the needed configuration data specified in path.config from pipelines.yml file.

Is it possible to use multiple pipelines in the Windows variant of Logstash? If so, how is it done? Would someone please instruct me on what I am doing wrong?

For what it is worth, when I start Logstash with the -f flag the pipelines.yml file is not loaded as expected, but Logstash does indeed start with the config that the pipelines file points to.

Here is that command line:

logstash.bat -f "C:\Program Files\Logstash\config\pipelines\dmarcxml.d\logstash.conf"

-Tony

You can't use drive letters in the path and I BELIEVE you have to use forward slashes....though backslashes may work too. Try the below instead.

path.config: "/Program Files/Logstash/config/pipelines/dmarcxml.d"

I implemented your suggestion, and modified the path.config line. We now look look like this.

  • pipeline.id: pipelinedmarcxml
    path.config: "/program files/logstash/config/pipelines/dmarcxml.d"
    pipeline.workers: 3

However, there was no change in behavior. I included the entire output.

[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Program Files/Logstash/modules/fb_apache/configuration"}
[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Program Files/Logstash/modules/netflow/configuration"}
[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"C:/Program Files/Logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.2-java/modules/arcsight/configuration"}
[WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.2"}
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[http://my_server:9200], bulk_path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", manage_template=>false, document_type=>"%{[@metadata][document_type]}", sniffing=>false, id=>"aadba5fcef4344a86d4ff81526835dbfa6bac63c4889322245f69daf37dec289", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1b63ba34-4400-4423-8ed9-0d7328eee692", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
[INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://my_server:9200/]}}
[INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://my_server:9200/, :path=>"/"}
[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://my_server:9200/"}
[INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://my_server:9200"]}
[INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://my_server:9200/]}}
[INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://my_server:9200/, :path=>"/"}
[WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://my_server:9200/"}
[INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>nil}
[WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x18cc8a run>"}
[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n", :backtrace=>["C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "C:/Program Files/Logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "C:/Program Files/Logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[INFO ][logstash.inputs.metrics ] Monitoring License OK
[INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x18cc8a run>"}

There are two interesting things in the log.

[WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

So, if you want Logstash to pay any attention to your pipelines.yml you shouldn't pass the path to a configuration file with -f.

LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n

This indicates that Logstash somehow picks up your jvm.options file (or whatever file the JVM-related text comes from). Where is that file? What other files are there in that directory?

[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n", :backtrace=>["C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "C:/Program Files/Logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "C:/Program Files/Logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

No sir, it's working now. It's a new problem with a config file on line 6, column 1. Did you set an incorrect value in your jvm.options file or are you trying to set JVM head sizes in your pipeline?

Edit: As Magnus initially caught, need to remove -f, which bypasses any pipeline config files when set. Just run logstash without any arguments.

Hey Magnus,

Yeah, I know that -f makes logstash ignore the pipelines.yml file.

If I use -f or not, I still get the message: [WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified.

For example I get that message with: logstash.bat or even logstash.bat -f "C:\Program Files\Logstash\config\pipelines\dmarcxml.d\logstash.conf".

I just mentioned the -f line in the original post to show that this command did in fact load Logstash even though pipelines was intentionally bypassed in that instance.

My goal is to get logstash to run without the -f, that way multiple pipelines can be used.

In regards to the jvm.options file, it is the out of box file with no changes. It is located here: C:\Program Files\Logstash\config.

These are the other files in the directory:
jvm.options
log42j.properties
logstash.yml
pipelines.yml
startup.options

Thanks for the sanity check!

-Tony

Hey wwalker,

The jvm.options has not been changed. It is what shipped with Windows logstash variant.

Yep, I want to run Logstash without the -f so, pipelines.yml is not ignored. I was just trying to point out that when I used the -f I could get Logstash to run. Did not mean to introduce confusion.

Here is want shows in the top of jvm.options file. I will admit, I have no clue what lines 6 and 7 do.

1## JVM configuration
2
3# Xms represents the initial size of total heap space
4# Xmx represents the maximum size of total heap space
5
6-Xms1g
7-Xmx1g
8
9################################################################
10## Expert settings
11################################################################
12##
13## All settings below this section are considered
14## expert settings. Don't tamper with them unless
15## you understand what you are doing
16##
17################################################################

Alright, walk us through your startup process. Do you have Logstash installed as a service or just running from a CMD prompt? Regardless of method, are you using any arguments after logstash? What's your pipeline configuration look like?

Does your jvm.optins file have the line numbering as well??

Logstash is running from a command prompt, but I will figure out how to install it as a service eventually. No arguments. From C:\Program Files\Logstash\Bin, I run: logstash.bat.

The pipeline.yml file is the default\shipped file with the exception of three lines.

  • pipeline.id: pipelinedmarcxml
    path.config: "/Program Files/Logstash/config/pipelines/dmarcxml.d"
    pipeline.workers: 3

I hope it is readable. I attached a screen shot of the modified section of the file.

Does the pipelines folder and dmacxml.d file exist? Looks like you are working off some of the things I did, lol.

No. I added them for sanity sake in the forum . Here is a screen shot of the beginning file.

jvm_options

-Tony

Yes, the pipelines folder and dmacxml.d folder exist, with a logstash.conf file in the dmarcxml.d folder.

pipelines_folder

dmarcxml

There's our problem. Point config.path to the actual file, not just the directory that it sits in.

Yea, testing it on Linux was much easier, but of course the data originates on windows :slight_smile:

I was excited for a second. However, including the file in the path.config did work. I wonder what I am missing - it has to be something silly at this point.

Here are the results from when I execute logstash.bat.

[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"C:/Program Files/Logstash/modules/fb_apache/configuration"}
[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"C:/Program Files/Logstash/modules/netflow/configuration"}
[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"C:/Program Files/Logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.2-java/modules/arcsight/configuration"}
[WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.2"}
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch hosts=>[http://my_server:9200], bulk_path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", manage_template=>false, document_type=>"%{[@metadata][document_type]}", sniffing=>false, id=>"aadba5fcef4344a86d4ff81526835dbfa6bac63c4889322245f69daf37dec289", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_c3275017-4945-4b7d-a280-2783bed76535", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
[INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http:/my_server:9200/]}}
[INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://my_server:9200/, :path=>"/"}
[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://my_server:9200/"}
[INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>nil}
[WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://my_server:9200"]}
[INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://my_server:9200/]}}
[INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://my_server:9200/, :path=>"/"}
[WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://my_server:9200/"}
[INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>nil}
[WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x15e4937 run>"}
[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n", :backtrace=>["C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "C:/Program Files/Logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "C:/Program Files/Logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "C:/Program Files/Logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "C:/Program Files/Logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "C:/Program Files/Logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
[INFO ][logstash.inputs.metrics ] Monitoring License OK
[INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x15e4937 run>"}

The Java JDK is installed?
JAVA_HOME system variable is set?
Are you running from an administrative command prompt?
Open your config files with Notepad++ and ensure the encoding is set to UTF-8 and NOT UTF-8-BOM.
I'm kinda grasping at straws here...not really sure where your problem is.

Try to set the path.config only to /config/pipelines/dmarcxml.d/logstash.conf
Now from the prompt cd to "program files/logstash" and try to start logstash with "bin/logstash"
Had a similar problem and it worked for me.

This morning, just tinkering with the files. I finally got Logstash to run with the logstash.bat command and use multiple pipelines. Unfortunately, I am not sure what the ah-ha fix was.

Here is the config in the pipelines.yml which is working:

  • pipeline.id: pipelinedmarcxml
    path.config: "/program files/logstash/config/pipelines/dmarcxml.d/dmarcpipeline.yml"
    pipeline.workers: 3

The odd thing is that this config did not seem to work until remarked and unremarked this line in the logstash.yml - which does not make sense, at least to me.

xpack.monitoring.elasticsearch.url: "http://my_server:9200"

Regardless, things work now. Yay!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.