First, I saw and tried the solution mentioned here and commented out path.config from /etc/logstash/logstash.yml
Looks like when I run logstash from command line all works well, but I need the logstash service to run. ie. "systemctl start logstash"
Logstash completely ignores pipelines.yml
Any advice or ideas?
Best,
JD
What are the contents of /etc/systemd/system/logstash.service?
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target
And pipelines.yml is in /etc/logstash?
Yes it is. When just executing "logstash" with no arguments it's reading pipelines.yml
[root@elk1 logstash]# cat pipelines.yml
- pipeline.id: test_vpctest
path.config: "/etc/logstash/conf.d/test_vpc_logstash.conf"
pipeline.workers: 3
[root@elk1 logstash]# pwd
/etc/logstash
[root@elk1 logstash]#
What's in the logs when you start it via systemctl?
It just keeps repeating those 2 lines, nothing else shows up:
[2017-11-15T16:07:58,670][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-11-15T16:07:58,675][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
Please report this in an issue at https://github.com/elastic/logstash/issues as this sounds like a bug.
Actually, you do have the path.settings command line option when starting it with systemd. I had exactly the same problem. My logs were saying:
Ignoring the 'pipelines.yml' file because modules or command line options are specified
So I removed the part with "--path.settings" "/etc/logstash" from /etc/systemd/system/logstash.service and now it does read my pipelines.yml file from /etc/logstash
Jonas, thank you this is an interesting idea. Do you mind sharing your logstash.service file ? I tried your option and currently there is no log written anywhere to see what's going on, but my logstash is not executing the content of pipelines.yml yet
Thank you,
JD
Hey JD, there you go :
[Unit]
Description=logstash
[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target
From my logs the last time Logstash was started:
[2017-12-18T12:17:55,757][INFO ][logstash.agent ] Pipelines running {:count=>3, :pipelines=>["indexing", "shipping", "aggregating"]}
I'm not sure if it's important but I stopped Logstash completely before modifying the service file. Of course, don't forget to reload the systemctl conf (sudo systemctl daemon-reload). Then just start it (sudo systemctl start logstash.service) and check if it's running properly (sudo systemctl status logstash.service). Logs should still go to /var/log/logstash (default for an RPM installation).
Hope this helps,
regards,
Jonas
Thank you for the detailed answer. Our logstash.service files are identical. My /etc/logstash/pipelines.yml looks like below and points to conf file locations under my "elastic" user
pipelines.yml
- pipeline.id: temp_vpctest_legacy path.config: "/home/elastic/logstashfiles/temp/temp_vpc/logstash_v2.conf" pipeline.workers: 3
- pipeline.id: dxc_vpctest_extended path.config: "/home/elastic/logstashfiles/temp/temp_vpc/logstash_v3_extended.conf" pipeline.workers: 3
The files are readable for everyone, and when I run "systemctl start logstash" it tells me it ignores pipelines.yml because I passed command line options. Maybe the bug is really a bug.
[2017-12-26T12:22:25,612][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2017-12-26T12:22:25,859][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/etc/logstash/conf.d/*.conf"}
My /etc/logstash/conf.d is empty, that's correct, but nothing is correct about passing arguments to logstash. I checked /etc/logstash/logstash.yml and other possible causes, and there is no argument passed that I can see. Weird.
I had the same issue, but there is definitely a solution. If i remember correctly, it was something like commenting out path.config in my logstash.yml. There is something in that config file that needs to be commented out to get it to work. Basically, everything in your config file is a flag that is passed to logstash when it is run as a service. You can take a look closer by looking at sudo systemctl status logstash.
Try commenting that out in your config and let me know if that works for you.
Here's the content of my /etc/logstash/pipelines.yml file:
- pipeline.id: shipping
path.config: "/etc/logstash/conf.d/shipping.conf"
- pipeline.id: indexing
path.config: "/etc/logstash/conf.d/indexing.conf"
- pipeline.id: aggregating
path.config: "/etc/logstash/conf.d/aggregating.conf"
May I suggest you remove everything but the 'required' settings from pipelines.yml (i.e. leave just the pipeline.id and path.config while respecting the format). I'd understood that it was important to have path.config on a new line with no dash (-) but I might be mistaken. I suppose the actual locations you're referring to in path.config don't matter as long as the config files are really there but it might be worth to try to move your files to /etc/logstash/conf.d to see if that would make a difference.
Also, moving at least one of your config files to /etc/logstash/conf.d would allow Logstash to actually start even if it keeps ignoring the pipelines.yml file and it would probably allow you to check if it's actually reading anything at all from the files in /etc/logstash (that's the default location for path.settings as per the documentation here)
Here's the content of my /etc/logstash directory:
$ ll /etc/logstash/
total 40
drwxrwxr-x 2 root root 4096 Dec 26 14:48 conf.d
drwxr-xr-x 3 root root 4096 Dec 11 10:29 jms
-rw-r--r-- 1 root root 1768 Dec 6 13:49 jvm.options
-rw-r--r-- 1 root root 1334 Nov 10 21:03 log4j2.properties
-rw-r--r-- 1 root root 6420 Dec 7 08:13 logstash.yml
-rw-r--r-- 1 root root 181 Dec 14 14:59 lookupHostname_agg.json
-rw-r--r-- 1 root root 181 Dec 6 13:33 lookupHostname.json
-rw-r--r-- 1 logstash logstash 234 Dec 14 14:55 pipelines.yml
-rw-r--r-- 1 root root 1659 Nov 10 21:03 startup.options
I just checked the startup.options file as well as it's used to generate the systemd service file and it'd also generated an environment file in /etc/default. You might want to have a look at it as it contains a variable : LS_SETTINGS_DIR="/etc/logstash"
And last but not least, @arisbanach is right in that the path.config in your logstash.yml should be commented out, otherwise Logstash will keep on ignoring the pipelines.yml file. I'd probably have to test now if it still works after putting back "--path.settings" "/etc/logstash" in the logstash.service systemd file...
Looks like you are correct. I moved on to a new problem ("No config files found in path") in pipelines.yml
But at least it is now reading pipelines.yml !
You were right because logstash.yml did carry settings from older versions. I cleaned anything but host and logs location, and pipelines.yml is now being read when I start logstash as a service.
THANK YOU!!!
now for that new problem 
"No config files found in path" I'll go do more research.
Glad to read that you solved the issue and good luck with the rest!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.