How to run configured pipeline.yml in logstash

Elastic Stack Logstash
1

I have configured 2 logstash.conf file in pipelines.yml file,

pipeline.id: sys_log_1
pipeline.workers: 2
path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_1.conf"
pipeline.id: sys_log_2
pipeline.workers: 2
path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_2.conf"

can anyone please tell me how can I run the pipelines.yml file.
I got struct in this.

2

Do you mean how to have logstash use your custom pipeline.yml file? See Multiple Pipelines | Logstash Reference [7.13] | Elastic on where to put it and what to name it.

3

Can you please provide me the
running command which I can use directly in terminal

4

How did you install logstash? tar.gz, deb, rpm, windows?
What Version?
What other configuration changes have you made?
What commands have you tried? Any? Did you get errors? If so share them.
Are you running from the command line or as a service?
Perhaps Share your logstash.yml

My point is ... if you want help, perhaps provide additional information then perhaps we can help.

In general if you made no other changes than that, if you start filebeat from the command line without the -f option it will read the pipelines.yml file and start those pipelines.

If you installed as a package and it is running as a service and you made little other changes that will also pick up the pipelines.yml

1 Like
5

Hi @stephenb

I downloaded as ZIP file
I'm using windows OS,

I have configured 2 logstash.conf file in pipelines.yml file,

pipeline.id: sys_log_1
pipeline.workers: 2
path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_1.conf"
pipeline.id: sys_log_2
pipeline.workers: 2
path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_2.conf"

Logstash version: Logstash/7.12.1

I haven't changed anything in logstash.yml file.
I updated only on the pipeline.yml file

ERROR LOGS:
D:\Elastic\Logstash\7.12.1>bin\logstash.bat
"Using bundled JDK: ""
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to D:/Elastic/Logstash/7.12.1/logs which is now configured via log4j2.properties
[2021-06-18T20:51:25,701][INFO ][logstash.runner ] Log4j configuration path used is: D:\Elastic\Logstash\7.12.1\config\log4j2.properties
[2021-06-18T20:51:25,716][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [mswin32-x86_64]"}
[2021-06-18T20:51:26,656][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-18T20:51:26,796][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2021-06-18T20:51:26,890][INFO ][logstash.runner ] Logstash shut down.
[2021-06-18T20:51:26,906][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at D_3a_.Elastic.Logstash.$7_dot_12_dot_1.lib.bootstrap.environment.(D:\Elastic\Logstash\7.12.1\lib\bootstrap\environment.rb:89) ~[?:?]

6

In logstash.yml file,

this is the only line enabled

pipeline.ordered: auto

I think this is the default value

7

@stephenb I have provided answers for your questions in the above chat,
Please take a look and let me know if you want any informations.

8

This is the key error

It indicates those paths you defined in the pipelines.yml are not found.. also You did not move the pipelines file did you?

Example, I'm sure you took a close look but could there be issues with upper or lower cases in the path.

path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_1.conf"

You can run the command with --debug and it should show you the paths it's actually trying to load and or give you some more detail.

Also If you try to run a a single conf file what happens

D:\Elastic\Logstash\7.12.1>bin\logstash.bat -f D:\Elastic\Logstash\7.12.1\config\logstash_sys_1.conf
9

Eventually I could be able to run pipeline.yml.

  1. Configured logstash_sys1.config
  2. In pipeline.yml

pipeline.id: sys_log_1
pipeline.workers: 2
path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_1.conf"

Instead I configured it,
path.config: "D:/Elastic/Logstash/7.12.1/config/logstash_sys_1.conf"

Then ran simply

bin\logstash

Now the logstash pipeline is working fine.

Thanks. :innocent:

1 Like
10

Glad it's working but I can't really tell what change was but I guess the path was not correct.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.