# Pipelines.yml ignored on logstash running as a service **URL:** \ **Category:** Logstash\ **Created:** [November 15, 2017, 9:19pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828 "2017-11-15T21:19:33Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [November 15, 2017, 9:19pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/1 "2017-11-15T21:19:33Z")
First, I saw and tried the solution mentioned [here](https://discuss.elastic.co/t/logstash-v6-beta-with-multiple-pipelines-configuration/102559) and commented out path.config from /etc/logstash/logstash.yml Looks like when I run logstash from command line all works well, but I need the logstash service to run. ie. "systemctl start logstash" Logstash completely ignores pipelines.yml Any advice or ideas? Best, JD ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [November 15, 2017, 9:34pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/2 "2017-11-15T21:34:13Z")
What are the contents of `/etc/systemd/system/logstash.service`? ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [November 15, 2017, 9:35pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/3 "2017-11-15T21:35:47Z")
[Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [November 15, 2017, 9:49pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/4 "2017-11-15T21:49:58Z")
And `pipelines.yml` is in `/etc/logstash`? ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [November 15, 2017, 9:50pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/5 "2017-11-15T21:50:47Z")
Yes it is. When just executing "logstash" with no arguments it's reading pipelines.yml [root@elk1 logstash]# cat pipelines.yml - [pipeline.id](http://pipeline.id): test\_vpctest path.config: "/etc/logstash/conf.d/test\_vpc\_logstash.conf" pipeline.workers: 3 [root@elk1 logstash]# pwd /etc/logstash [root@elk1 logstash]# ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [November 15, 2017, 10:05pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/6 "2017-11-15T22:05:41Z")
What's in the logs when you start it via `systemctl`? ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [November 15, 2017, 10:08pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/7 "2017-11-15T22:08:25Z")
It just keeps repeating those 2 lines, nothing else shows up: [2017-11-15T16:07:58,670][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} [2017-11-15T16:07:58,675][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [November 15, 2017, 10:09pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/8 "2017-11-15T22:09:27Z")
Please report this in an issue at [https://github.com/elastic/logstash/issues](https://github.com/elastic/logstash/issues) as this sounds like a bug. ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [November 15, 2017, 10:15pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/9 "2017-11-15T22:15:04Z")
Done. Issue opened [here](https://github.com/elastic/logstash/issues/8684) Thanks for looking at it! ---
**Author:** ![Jon\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jon_d/32/25657_2.png) [@Jon\_D](https://discuss.elastic.co/u/Jon_D)\ **Post date:** [December 7, 2017, 8:07am UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/10 "2017-12-07T08:07:57Z")
Actually, you do have the path.settings command line option when starting it with systemd. I had exactly the same problem. My logs were saying: ``` Ignoring the 'pipelines.yml' file because modules or command line options are specified ``` So I removed the part with `"--path.settings" "/etc/logstash"` from `/etc/systemd/system/logstash.service` and now it does read my `pipelines.yml` file from `/etc/logstash` ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [December 22, 2017, 4:39pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/11 "2017-12-22T16:39:24Z")
Jonas, thank you this is an interesting idea. Do you mind sharing your logstash.service file ? I tried your option and currently there is no log written anywhere to see what's going on, but my logstash is not executing the content of pipelines.yml yet Thank you, JD ---
**Author:** ![Jon\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jon_d/32/25657_2.png) [@Jon\_D](https://discuss.elastic.co/u/Jon_D)\ **Post date:** [December 26, 2017, 11:30am UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/12 "2017-12-26T11:30:11Z")
Hey JD, there you go : ``` [Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 [Install] WantedBy=multi-user.target ``` From my logs the last time Logstash was started: ``` [2017-12-18T12:17:55,757][INFO][logstash.agent] Pipelines running {:count=>3, :pipelines=>["indexing", "shipping", "aggregating"]} ``` I'm not sure if it's important but I stopped Logstash completely before modifying the service file. Of course, don't forget to reload the systemctl conf (`sudo systemctl daemon-reload`). Then just start it (`sudo systemctl start logstash.service`) and check if it's running properly (`sudo systemctl status logstash.service`). Logs should still go to /var/log/logstash (default for an RPM installation). Hope this helps, regards, Jonas ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [December 26, 2017, 5:28pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/13 "2017-12-26T17:28:37Z")
Thank you for the detailed answer. Our logstash.service files are identical. My /etc/logstash/pipelines.yml looks like below and points to conf file locations under my "elastic" user **pipelines.yml** `- pipeline.id: temp_vpctest_legacy path.config: "/home/elastic/logstashfiles/temp/temp_vpc/logstash_v2.conf" pipeline.workers: 3` `- pipeline.id: dxc_vpctest_extended path.config: "/home/elastic/logstashfiles/temp/temp_vpc/logstash_v3_extended.conf" pipeline.workers: 3` The files are readable for everyone, and when I run "systemctl start logstash" it tells me it ignores pipelines.yml because I passed command line options. Maybe the bug is really a bug. ``` [2017-12-26T12:22:25,612][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2017-12-26T12:22:25,859][INFO][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"} ``` My /etc/logstash/conf.d is empty, that's correct, but nothing is correct about passing arguments to logstash. I checked /etc/logstash/logstash.yml and other possible causes, and there is no argument passed that I can see. Weird. ---
**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\ **Post date:** [December 27, 2017, 1:25am UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/14 "2017-12-27T01:25:09Z")
I had the same issue, but there is definitely a solution. If i remember correctly, it was something like commenting out `path.config` in my `logstash.yml`. There is something in that config file that needs to be commented out to get it to work. Basically, everything in your config file is a flag that is passed to logstash when it is run as a service. You can take a look closer by looking at `sudo systemctl status logstash`. Try commenting that out in your config and let me know if that works for you. ---
**Author:** ![Jon\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jon_d/32/25657_2.png) [@Jon\_D](https://discuss.elastic.co/u/Jon_D)\ **Post date:** [December 27, 2017, 8:25am UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/15 "2017-12-27T08:25:00Z")
Here's the content of my `/etc/logstash/pipelines.yml` file: ``` - pipeline.id: shipping path.config: "/etc/logstash/conf.d/shipping.conf" - pipeline.id: indexing path.config: "/etc/logstash/conf.d/indexing.conf" - pipeline.id: aggregating path.config: "/etc/logstash/conf.d/aggregating.conf" ``` May I suggest you remove everything but the 'required' settings from `pipelines.yml` (i.e. leave just the `pipeline.id` and `path.config` while respecting the format). I'd understood that it was important to have path.config on a new line with no dash (-) but I might be mistaken. I suppose the actual locations you're referring to in path.config don't matter as long as the config files are really there but it might be worth to try to move your files to `/etc/logstash/conf.d` to see if that would make a difference. Also, moving at least one of your config files to `/etc/logstash/conf.d` would allow Logstash to actually start even if it keeps ignoring the `pipelines.yml` file and it would probably allow you to check if it's actually reading anything at all from the files in `/etc/logstash` (that's the default location for `path.settings` as per the documentation [here](https://www.elastic.co/guide/en/logstash/current/dir-layout.html#deb-layout)) Here's the content of my `/etc/logstash` directory: ``` $ ll /etc/logstash/ total 40 drwxrwxr-x 2 root root 4096 Dec 26 14:48 conf.d drwxr-xr-x 3 root root 4096 Dec 11 10:29 jms -rw-r--r-- 1 root root 1768 Dec 6 13:49 jvm.options -rw-r--r-- 1 root root 1334 Nov 10 21:03 log4j2.properties -rw-r--r-- 1 root root 6420 Dec 7 08:13 logstash.yml -rw-r--r-- 1 root root 181 Dec 14 14:59 lookupHostname_agg.json -rw-r--r-- 1 root root 181 Dec 6 13:33 lookupHostname.json -rw-r--r-- 1 logstash logstash 234 Dec 14 14:55 pipelines.yml -rw-r--r-- 1 root root 1659 Nov 10 21:03 startup.options ``` I just checked the `startup.options` file as well as it's used to generate the systemd service file and it'd also generated an environment file in `/etc/default`. You might want to have a look at it as it contains a variable : `LS_SETTINGS_DIR="/etc/logstash"` ---
**Author:** ![Jon\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jon_d/32/25657_2.png) [@Jon\_D](https://discuss.elastic.co/u/Jon_D)\ **Post date:** [December 29, 2017, 1:01pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/16 "2017-12-29T13:01:02Z")
And last but not least, @arisbanach is right in that the `path.config` in your `logstash.yml` should be commented out, otherwise Logstash will keep on ignoring the `pipelines.yml` file. I'd probably have to test now if it still works after putting back "`--path.settings" "/etc/logstash`" in the `logstash.service` systemd file... ---
**Author:** ![dorj1234](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dorj1234/32/21339_2.png) [@dorj1234](https://discuss.elastic.co/u/dorj1234)\ **Post date:** [January 2, 2018, 7:20pm UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/17 "2018-01-02T19:20:16Z")
Looks like you are correct. I moved on to a new problem ("No config files found in path") in pipelines.yml But at least it is now reading pipelines.yml ! You were right because logstash.yml did carry settings from older versions. I cleaned anything but host and logs location, and pipelines.yml is now being read when I start logstash as a service. THANK YOU!!! now for that new problem 🙂 "No config files found in path" I'll go do more research. ---
**Author:** ![Jon\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jon_d/32/25657_2.png) [@Jon\_D](https://discuss.elastic.co/u/Jon_D)\ **Post date:** [January 8, 2018, 6:24am UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/18 "2018-01-08T06:24:15Z")
Glad to read that you solved the issue and good luck with the rest! 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2018, 6:24am UTC](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828/19 "2018-02-05T06:24:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.