# Pipelines.yml ignored on logstash **URL:** \ **Category:** Logstash\ **Created:** [September 20, 2022, 7:41pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807 "2022-09-20T19:41:10Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 20, 2022, 7:41pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/1 "2022-09-20T19:41:10Z")
I have looked at the following: > [@Ignoring the 'pipelines.yml' file because modules or command line options are specified - RPM](https://discuss.elastic.co/t/ignoring-the-pipelines-yml-file-because-modules-or-command-line-options-are-specified-rpm/183795): > > i have installed logstash through RPM on my ec2 instance and started \< ./logstash --path.settings /etc/logstash/ --path.config /etc/logstash/ /\> but i see below error. Can someone help out here? [root@ip-10-1-1-133 bin]# ./logstash --path.settings /etc/logstash/ --path.config /etc/logstash/ -e OpenJDK 64-Bit Server VM warning: If the number of processors is expected to increase from one, then you should configure the number of parallel GC threads appropriately using -XX:ParallelGCThreads=N … > > [@Logstash v6 beta with multiple pipelines configuration](https://discuss.elastic.co/t/logstash-v6-beta-with-multiple-pipelines-configuration/102559/2): > > Note, in above output there was only one pipeline started for two config files which are located in /etc/logstash/conf.d/ dir. pipelines.yml seems to have been ignored. > [@Pipelines.yml ignored on logstash running as a service](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828): > > First, I saw and tried the solution mentioned [here](https://discuss.elastic.co/t/logstash-v6-beta-with-multiple-pipelines-configuration/102559) and commented out path.config from /etc/logstash/logstash.yml Looks like when I run logstash from command line all works well, but I need the logstash service to run. ie. "systemctl start logstash" Logstash completely ignores pipelines.yml Any advice or ideas? Best, JD I have LS 8.3.3 and here is version 1 of pipelines.yml located in /opt/logstash/config/pipelines.yml ```auto - pipeline.id: main path.config: "/opt/logstash/lab_conf/*.conf" - pipeline.id: elastic-processing path.config: "/opt/logstash/elastic_conf/*.conf" - pipeline.id: qradar-processing path.config: "/opt/logstash/qradar_conf/*.conf" ``` Here is /opt/logstash/config/logstash.yml ```auto node.name: logstash1 path.data: /opt/logstash/data #path.config: /opt/logstash/lab_conf ``` when I run with this config, here is the result: ```auto root@logstash1:/opt/logstash# bin/logstash -t Using bundled JDK: /opt/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties [2022-09-20T19:25:27,163][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash/config/log4j2.properties [2022-09-20T19:25:27,167][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"} [2022-09-20T19:25:27,169][INFO][logstash.runner] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] ERROR: Failed to read pipelines yaml file. Location: /opt/logstash/config/pipelines.yml usage: bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG] bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG] bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG] bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace] bin/logstash -V [--log.level fatal|error|warn|info|debug|trace] bin/logstash --help [2022-09-20T19:25:27,389][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?] at opt.logstash.lib.bootstrap.environment.
(/opt/logstash/lib/bootstrap/environment.rb:91) ~[?:?] ``` If I uncomment the path.config in logstash.yml, it still doesn't read the pipelines.yml file: ```auto root@logstash1:/opt/logstash# bin/logstash -t Using bundled JDK: /opt/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Sending Logstash logs to /opt/logstash/logs which is now configured via log4j2.properties [2022-09-20T19:38:01,975][INFO][logstash.runner] Log4j configuration path used is: /opt/logstash/config/log4j2.properties [2022-09-20T19:38:01,979][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.3.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.15+10 on 11.0.15+10 +indy +jit [linux-x86_64]"} [2022-09-20T19:38:01,981][INFO][logstash.runner] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED] [2022-09-20T19:38:02,170][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2022-09-20T19:38:02,534][INFO][org.reflections.Reflections] Reflections took 60 ms to scan 1 urls, producing 124 keys and 408 values [2022-09-20T19:38:02,828][INFO][logstash.javapipeline] Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise. Configuration OK [2022-09-20T19:38:02,829][INFO][logstash.runner] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash ``` I have also tried naming the first pipeline something else besides 'main' and that doesn't work either. Suggestions? ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 20, 2022, 7:43pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/2 "2022-09-20T19:43:23Z")
Version 2 of the pipelines.yml just moves all the config files into the pipelines.yml file using config.string. For example: ```auto - pipeline.id: main config.string: | input { tcp { port => 1514 }} output { pipeline { send_to => elastic } pipeline { send_to => qradar } } ``` Which just makes a long pipelines.yml file but doesn't fix the problem... ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 20, 2022, 7:54pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/3 "2022-09-20T19:54:56Z")
> [@reswob](#): > > ERROR: Failed to read pipelines yaml file. Location: /opt/logstash/config/pipelines.yml It is failing to read the file, most likely a permissions error. > [@](#): > > If I uncomment the path.config in logstash.yml, it still doesn't read the pipelines.yml file: That is expected. You can use path.confiig or pipelines.yml, you cannot use both. ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 20, 2022, 8:03pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/4 "2022-09-20T20:03:15Z")
perms are: ```auto -rw-r--r-- 1 logstash logstash 6414 Sep 20 19:13 pipelines.yml ``` These are the same perms for all the files in the /opt/logstash/config folder. There were root.root, but I was trying different things to get it to work..... should I change them back? (sorry for the noob question....) This was installed using the tarball... not rpm or deb... ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 20, 2022, 9:25pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/5 "2022-09-20T21:25:45Z")
Does the logstash user has permissions to the config path as well? If you want to use `pipelines.yml`, remove the `paht.config` line from `logstash.yml`. Also, try to use `--path.settings` in the command line pointing to the folder where the `logstash.yml` and `pipelines.yml` file are stored. `bin/logstash --path.settings /opt/logstash/config` should start logstash using the `pipelines.yml`. ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 20, 2022, 9:38pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/6 "2022-09-20T21:38:53Z")
Will check 1st thing tomorrow.... ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 21, 2022, 12:46pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/7 "2022-09-21T12:46:06Z")
OK, I removed the path.config from the logstash.yml su to logstash, checked permissions. As logstash, I could read logstash.yml and pipelines.yml exited to root, then tried to run /opt/logstash/bin/logstash as root. same error. tried to run as root using /opt/logstash/bin/logstash --path.settings /opt/logstash/config. same error. chowned entire /opt/logstash directory to logstash.logstash and tried to run as root. same error. su to logstash and tried to run as logstash. same error. Still able to read both files as logstash user. ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 21, 2022, 12:58pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/8 "2022-09-21T12:58:24Z")
OK. I just removed the pipelines.yml file from the config directory, then su'd to the logstash user and created a new pipelines.yml file and started logstash as the logstash user. That worked. from the log: ...snip... Pipelines running {:count=\>3, :running\_pipelines=\>[:main, :"elastic-processing", :"qradar-processing"], :non\_running\_pipelines=\>} ...snip... (I got a bunch of pipeline errors, but that is a different problem I think) Works when I run as root as well. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [September 21, 2022, 1:09pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/9 "2022-09-21T13:09:20Z")
> [@reswob](#): > > created a new pipelines.yml file and started logstash as the logstash user. This could mean that your old `pipelines.yml` had some formatting error, which is one thing that could give you the error you were getting before. > [@reswob](#): > > Works when I run as root as well. I would say that you should avoid running logstash as root even for testing, when logstash runs it creates some temporary files and paths depending on your pipeline, if you run it as a root, it will create them as the root user, which could give you permissions errors when you run as the logstash user. ---
**Author:** ![reswob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/reswob/32/38015_2.png) [@reswob](https://discuss.elastic.co/u/reswob)\ **Post date:** [September 21, 2022, 1:17pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/10 "2022-09-21T13:17:28Z")
Got it. Makes sense why I should not run as root. (besides other good reasons) Thanks for the help. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 19, 2022, 1:18pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash/314807/11 "2022-10-19T13:18:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.