# Unable to run Logstash 7.9.2 as a service on Ubuntu 18.04 **URL:** \ **Category:** Logstash\ **Created:** [October 5, 2020, 2:15pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005 "2020-10-05T14:15:35Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 5, 2020, 2:15pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/1 "2020-10-05T14:15:36Z")
I'm getting a warning and an error when I try to start logstash with `systemctl start logstash`: ```auto [2020-10-05T14:01:19,067][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified [2020-10-05T14:01:19,670][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 6, column 1 (byte 132) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]} ``` [Pipelines.yml ignored on logstash running as a service - Elastic Stack / Logstash - Discuss the Elastic Stack](https://discuss.elastic.co/t/pipelines-yml-ignored-on-logstash-running-as-a-service/107828) suggests changing `/etc/systemd/system/logstash.service` so that `--path.settings` is no longer specified. I've done that and this is the revised service file: ```auto [Unit] Description=logstash [Service] Type=simple User=logstash Group=logstash # Load env vars from /etc/default/ and /etc/sysconfig/ if they exist. # Prefixing the path with '-' makes it try to load, but if the file doesn't # exist, it continues onward. EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash ExecStart=/usr/share/logstash/bin/logstash # "--path.settings" "/etc/logstash" Restart=always WorkingDirectory=/ Nice=19 LimitNOFILE=16384 # When stopping, how long to wait before giving up and sending SIGKILL? # Keep in mind that SIGKILL on a process can cause data loss. TimeoutStopSec=infinity [Install] WantedBy=multi-user.target ``` The error is particularly puzzling. The only reference I can find to a pipeline ID called main is in the default `pipelines.yml` **which is apparently being ignored.** I've nevertheless changed `pipelines.yml` to read thus: ```auto # This file is where you define your pipelines. You can define multiple. # For more information on multiple pipelines, see the documentation: # https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html - pipeline.id: auth0 path.config: "/etc/logstash/conf.d/auth0.conf" ``` where `auth0.conf` is (after redaction): ```auto input { http { id => "REDACTED" user => "REDACTED" password => "REDACTED" port => 9000 ssl => true ssl_certificate => "REDACTED" ssl_key => "REDACTED" } } filter { mutate { rename => { "_id" => "log_id" } } } output { amazon_es { hosts => ["REDACTED"] region => "us-east-1" index => "REDACTED" } stdout {} } ``` If I run logstash manually with: `/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d` then it works. There isn't a detected error within `auth0.conf`. ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 6, 2020, 7:21am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/2 "2020-10-06T07:21:04Z")
Running `/usr/share/logstash/bin/logstash -t --log.level=debug` reveals some interesting things ... Bear in mind that this installation of Logstash has been done via the instructions at [Installing Logstash | Logstash Reference [7.9] | Elastic](https://www.elastic.co/guide/en/logstash/7.9/installing-logstash.html#_apt). 1. The default for `path.settings` is `/usr/share/logstash/config` **which does not exist**. Instead, the installation has put the config into `/etc/logstash` but hasn't configured Logstash to use that. 2. If I add `--path.settings=/etc/logstash` to the test command, I start to get some light into what is causing the ConfigurationError exception ... it is trying to read `/etc/logstash/startup.options` which isn't a YAML file and, according to the comments in the file "These settings are ONLY used by $LS\_HOME/bin/system-install ... and is not used by Logstash itself". So how do I configure the Logstash service correctly so that it starts and, more importantly, why isn't this being done when installed per the official instructions? ---
**Author:** ![abertolli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abertolli/32/76550_2.png) [@abertolli](https://discuss.elastic.co/u/abertolli)\ **Post date:** [October 6, 2020, 7:44am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/3 "2020-10-06T07:44:04Z")
Is path.config set in logstash.yml? If so, then it will ignore pipelines.yml. This is actually OK (hence the warning). But when it goes to read the file indicated in logstash.yml, it doesn't find a valid pipeline (hence the error). ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 6, 2020, 7:57am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/4 "2020-10-06T07:57:56Z")
> [@abertolli](#): > > Is path.config set in logstash.yml? Yes, it is. OK, so I can ignore the warning ... but I'm still hitting the fact that logstash then tries to read `startup.options` because it is in /etc/logstash. At the moment, the only workaround I seem to be able to use is to get logstash to look at /etc/logstash/conf.d instead, so that it ignores everything in /etc/logstash. It just seems strange to me that the built package isn't working "out of the box" and, as a result, I'm left not knowing what the "right" thing is to do to get it working. ---
**Author:** ![abertolli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abertolli/32/76550_2.png) [@abertolli](https://discuss.elastic.co/u/abertolli)\ **Post date:** [October 7, 2020, 4:06pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/5 "2020-10-07T16:06:02Z")
Therefore, if path.config is set in logstash.yml, and it's set to something other than auth0.conf, it's not reading auth0.conf... ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 8, 2020, 6:52am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/6 "2020-10-08T06:52:41Z")
> [@abertolli](#): > > Therefore, if path.config is set in logstash.yml, and it's set to something other than auth0.conf, it's not reading auth0.conf... That doesn't make a lot of sense, I'm afraid. Out of the box, `/etc/logstash/pipelines.yml` is set to specify `path.config` as `/etc/logstash/conf.d/*.conf`, implying (correctly, I hope) that Logstash would then read all of the config files in that directory. However, out of the box, `pipelines.yml` is ignored when Logstash is run as a service. This seems to be in part due to the fact that Logstash is configured, by default, to load configuration from `/usr/share/logstash/config`. One workaround I'm testing is to symlink from there to `/etc/logstash` and then, in `/etc/logstash/logstash.yml`, change `path.config` to `/etc/logstash/conf.d` in order to avoid the error generated by Logstash trying to parse `/etc/logstash/startup.options`. All in all, the default Debian/Ubuntu installation of Logstash is a mess. It does not work and it needs fixing. ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 8, 2020, 7:30am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/7 "2020-10-08T07:30:20Z")
Do you mind to share what is your OS and how did you install the logstash? ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 8, 2020, 7:33am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/8 "2020-10-08T07:33:58Z")
> [@kavierkoo](#): > > Do you mind to share what is your OS and how did you install the logstash? It is Ubuntu 18.04. I followed the instructions in [Installing Logstash | Logstash Reference [7.9] | Elastic](https://www.elastic.co/guide/en/logstash/7.9/installing-logstash.html#_apt) ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 8, 2020, 8:40am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/9 "2020-10-08T08:40:21Z")
Just for checking sake, can you share the output of `ls -la /etc/logstash` ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 8, 2020, 9:05am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/10 "2020-10-08T09:05:14Z")
> [@kavierkoo](#): > > Just for checking sake, can you share the output of `ls -la /etc/logstash` ```auto $ ls -la /etc/logstash total 24 drwxrwxr-x 1 root root 4096 Oct 8 07:48 . drwxr-xr-x 1 root root 4096 Oct 8 10:04 .. drwxrwxr-x 1 root root 4096 Sep 23 04:13 conf.d -rw-r--r-- 1 root root 2019 Sep 23 04:10 jvm.options -rw-r--r-- 1 root root 9097 Sep 23 04:10 log4j2.properties -rw-r--r-- 1 root root 342 Sep 23 04:10 logstash-sample.conf -rw-r--r-- 1 root root 10693 Oct 8 07:48 logstash.yml -rw-r--r-- 1 root root 285 Sep 23 04:10 pipelines.yml -rw------- 1 root root 1696 Sep 23 04:10 startup.options ``` ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 8, 2020, 9:23am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/11 "2020-10-08T09:23:12Z")
Sorry, can you share t he output of `ls -laR /etc/logstash` once again with recursive? ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 8, 2020, 9:36am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/12 "2020-10-08T09:36:44Z")
> [@kavierkoo](#): > > Sorry, can you share t he output of `ls -laR /etc/logstash` once again with recursive? The following is on a **clean** installation of Logstash: ```auto $ ls -laR /etc/logstash /etc/logstash: total 24 drwxrwxr-x 1 root root 4096 Oct 8 07:48 . drwxr-xr-x 1 root root 4096 Oct 8 10:34 .. drwxrwxr-x 1 root root 4096 Sep 23 04:13 conf.d -rw-r--r-- 1 root root 2019 Sep 23 04:10 jvm.options -rw-r--r-- 1 root root 9097 Sep 23 04:10 log4j2.properties -rw-r--r-- 1 root root 342 Sep 23 04:10 logstash-sample.conf -rw-r--r-- 1 root root 10693 Oct 8 07:48 logstash.yml -rw-r--r-- 1 root root 285 Sep 23 04:10 pipelines.yml -rw------- 1 root root 1696 Sep 23 04:10 startup.options /etc/logstash/conf.d: total 0 drwxrwxr-x 1 root root 4096 Sep 23 04:13 . drwxrwxr-x 1 root root 4096 Oct 8 07:48 .. ``` On the system I've been working on, I've got a single `.conf` file in `/etc/logstash/conf.d`. I've also been "fiddling" with the various files in `/etc/logstash` so the datestamps would be newer. ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 8, 2020, 12:27pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/13 "2020-10-08T12:27:58Z")
I'm trying to do a fresh install of logstash on my ubuntu server to see if I could simulate the same problem. But my environment work as expected and logstash start with no issue. Below are codes I ran: java & logstash installation ```auto apt install default-jdk wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add - sudo apt-get install apt-transport-https echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list sudo apt-get update && sudo apt-get install logstash ``` modify `pipelines.yml` ```auto vi /etc/logstash/pipelines.yml - pipeline.id: test path.config: "/etc/logstash/conf.d/test.conf" ``` create `test.conf` ```auto vi /etc/logstash/conf.d/test.conf input { stdin { } } output { stdout {} } ``` Start service ```auto systemctl start logstash journalctl -xe ``` Logs ```auto Oct 08 05:15:39 ubuntu logstash[2898]: [2020-10-08T05:15:39,614][INFO][logstash.javapipeline][test] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/test.conf"], :thread=>"#1.04} Oct 08 05:15:40 ubuntu logstash[2898]: [2020-10-08T05:15:40,702][INFO][logstash.javapipeline][test] Pipeline started {"pipeline.id"=>"test"} Oct 08 05:15:40 ubuntu logstash[2898]: [2020-10-08T05:15:40,941][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600} ``` I have no idea why you are having this issue too. Apologize that I couldn't able to help. ---
**Author:** ![Philip\_Colmer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/philip_colmer/32/50355_2.png) [@Philip\_Colmer](https://discuss.elastic.co/u/Philip_Colmer)\ **Post date:** [October 8, 2020, 1:36pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/14 "2020-10-08T13:36:44Z")
> [@kavierkoo](#): > > I have no idea why you are having this issue too. > Apologize that I couldn't able to help. Actually, you have helped. I tried installing it again on a clean system and, like you, it worked. So I purged it from this system and reinstalled it and now it is working. So, thank you. I have no idea what has changed. Maybe I fiddled with too many files without remembering 🙂 ---
**Author:** ![kavierkoo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kavierkoo/32/86555_2.png) [@kavierkoo](https://discuss.elastic.co/u/kavierkoo)\ **Post date:** [October 8, 2020, 2:06pm UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/15 "2020-10-08T14:06:20Z")
haha.. oh well glad that I could help then!! ---
**Author:** ![abertolli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abertolli/32/76550_2.png) [@abertolli](https://discuss.elastic.co/u/abertolli)\ **Post date:** [October 9, 2020, 7:06am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/16 "2020-10-09T07:06:44Z")
I'm glad it's working for you. But I believe your problem was that you somehow had path.config set in your logstash.yml file. And this was fixed because you started from scratch with a default set of config files. Of course it's impossible to tell now. But for any future folks having a similar problem, I'll spell it out: Logstash reads logstash.yml first. If you have path.config set, you will not be using pipelines.yml. Check logstash.yml first, to see if it's loading a different pipeline than the default. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 6, 2020, 7:06am UTC](https://discuss.elastic.co/t/unable-to-run-logstash-7-9-2-as-a-service-on-ubuntu-18-04/251005/17 "2020-11-06T07:06:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.