Pipelines.yml ignored on logstash running as a service

First, I saw and tried the solution mentioned here and commented out path.config from /etc/logstash/logstash.yml

Looks like when I run logstash from command line all works well, but I need the logstash service to run. ie. "systemctl start logstash"
Logstash completely ignores pipelines.yml

Any advice or ideas?

1 Like

What are the contents of /etc/systemd/system/logstash.service?


ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"


And pipelines.yml is in /etc/logstash?

Yes it is. When just executing "logstash" with no arguments it's reading pipelines.yml

[root@elk1 logstash]# cat pipelines.yml
- pipeline.id: test_vpctest
path.config: "/etc/logstash/conf.d/test_vpc_logstash.conf"
pipeline.workers: 3
[root@elk1 logstash]# pwd
[root@elk1 logstash]#

What's in the logs when you start it via systemctl?

It just keeps repeating those 2 lines, nothing else shows up:

[2017-11-15T16:07:58,670][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-15T16:07:58,675][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}

Please report this in an issue at https://github.com/elastic/logstash/issues as this sounds like a bug.

Done. Issue opened here
Thanks for looking at it!

Actually, you do have the path.settings command line option when starting it with systemd. I had exactly the same problem. My logs were saying:

Ignoring the 'pipelines.yml' file because modules or command line options are specified

So I removed the part with "--path.settings" "/etc/logstash" from /etc/systemd/system/logstash.service and now it does read my pipelines.yml file from /etc/logstash

Jonas, thank you this is an interesting idea. Do you mind sharing your logstash.service file ? I tried your option and currently there is no log written anywhere to see what's going on, but my logstash is not executing the content of pipelines.yml yet

Thank you,

Hey JD, there you go :


# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.


From my logs the last time Logstash was started:

[2017-12-18T12:17:55,757][INFO ][logstash.agent           ] Pipelines running {:count=>3, :pipelines=>["indexing", "shipping", "aggregating"]}

I'm not sure if it's important but I stopped Logstash completely before modifying the service file. Of course, don't forget to reload the systemctl conf (sudo systemctl daemon-reload). Then just start it (sudo systemctl start logstash.service) and check if it's running properly (sudo systemctl status logstash.service). Logs should still go to /var/log/logstash (default for an RPM installation).

Hope this helps,

Thank you for the detailed answer. Our logstash.service files are identical. My /etc/logstash/pipelines.yml looks like below and points to conf file locations under my "elastic" user

- pipeline.id: temp_vpctest_legacy path.config: "/home/elastic/logstashfiles/temp/temp_vpc/logstash_v2.conf" pipeline.workers: 3
- pipeline.id: dxc_vpctest_extended path.config: "/home/elastic/logstashfiles/temp/temp_vpc/logstash_v3_extended.conf" pipeline.workers: 3

The files are readable for everyone, and when I run "systemctl start logstash" it tells me it ignores pipelines.yml because I passed command line options. Maybe the bug is really a bug.

[2017-12-26T12:22:25,612][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2017-12-26T12:22:25,859][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}

My /etc/logstash/conf.d is empty, that's correct, but nothing is correct about passing arguments to logstash. I checked /etc/logstash/logstash.yml and other possible causes, and there is no argument passed that I can see. Weird.

I had the same issue, but there is definitely a solution. If i remember correctly, it was something like commenting out path.config in my logstash.yml. There is something in that config file that needs to be commented out to get it to work. Basically, everything in your config file is a flag that is passed to logstash when it is run as a service. You can take a look closer by looking at sudo systemctl status logstash.

Try commenting that out in your config and let me know if that works for you.


Here's the content of my /etc/logstash/pipelines.yml file:

- pipeline.id: shipping
  path.config: "/etc/logstash/conf.d/shipping.conf"
- pipeline.id: indexing
  path.config: "/etc/logstash/conf.d/indexing.conf"
- pipeline.id: aggregating
  path.config: "/etc/logstash/conf.d/aggregating.conf"

May I suggest you remove everything but the 'required' settings from pipelines.yml (i.e. leave just the pipeline.id and path.config while respecting the format). I'd understood that it was important to have path.config on a new line with no dash (-) but I might be mistaken. I suppose the actual locations you're referring to in path.config don't matter as long as the config files are really there but it might be worth to try to move your files to /etc/logstash/conf.d to see if that would make a difference.

Also, moving at least one of your config files to /etc/logstash/conf.d would allow Logstash to actually start even if it keeps ignoring the pipelines.yml file and it would probably allow you to check if it's actually reading anything at all from the files in /etc/logstash (that's the default location for path.settings as per the documentation here)

Here's the content of my /etc/logstash directory:

$ ll /etc/logstash/
total 40
drwxrwxr-x 2 root     root     4096 Dec 26 14:48 conf.d
drwxr-xr-x 3 root     root     4096 Dec 11 10:29 jms
-rw-r--r-- 1 root     root     1768 Dec  6 13:49 jvm.options
-rw-r--r-- 1 root     root     1334 Nov 10 21:03 log4j2.properties
-rw-r--r-- 1 root     root     6420 Dec  7 08:13 logstash.yml
-rw-r--r-- 1 root     root      181 Dec 14 14:59 lookupHostname_agg.json
-rw-r--r-- 1 root     root      181 Dec  6 13:33 lookupHostname.json
-rw-r--r-- 1 logstash logstash  234 Dec 14 14:55 pipelines.yml
-rw-r--r-- 1 root     root     1659 Nov 10 21:03 startup.options

I just checked the startup.options file as well as it's used to generate the systemd service file and it'd also generated an environment file in /etc/default. You might want to have a look at it as it contains a variable : LS_SETTINGS_DIR="/etc/logstash"

1 Like

And last but not least, @arisbanach is right in that the path.config in your logstash.yml should be commented out, otherwise Logstash will keep on ignoring the pipelines.yml file. I'd probably have to test now if it still works after putting back "--path.settings" "/etc/logstash" in the logstash.service systemd file...

1 Like

Looks like you are correct. I moved on to a new problem ("No config files found in path") in pipelines.yml

But at least it is now reading pipelines.yml !
You were right because logstash.yml did carry settings from older versions. I cleaned anything but host and logs location, and pipelines.yml is now being read when I start logstash as a service.


now for that new problem :slight_smile:
"No config files found in path" I'll go do more research.

1 Like

Glad to read that you solved the issue and good luck with the rest! :slight_smile:

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.