Unable to run Logstash 7.9.2 as a service on Ubuntu 18.04

Elastic Stack Logstash
1

I'm getting a warning and an error when I try to start logstash with systemctl start logstash:

[2020-10-05T14:01:19,067][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-10-05T14:01:19,670][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"input\", \"filter\", \"output\" at line 6, column 1 (byte 132) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in `block in converge_state'"]}

Pipelines.yml ignored on logstash running as a service - Elastic Stack / Logstash - Discuss the Elastic Stack suggests changing /etc/systemd/system/logstash.service so that --path.settings is no longer specified. I've done that and this is the revised service file:

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash
# "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

The error is particularly puzzling. The only reference I can find to a pipeline ID called main is in the default pipelines.yml which is apparently being ignored. I've nevertheless changed pipelines.yml to read thus:

# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

- pipeline.id: auth0
  path.config: "/etc/logstash/conf.d/auth0.conf"

where auth0.conf is (after redaction):

input {
        http {
                id => "REDACTED"
                user => "REDACTED"
                password => "REDACTED"
                port => 9000
                ssl => true
                ssl_certificate => "REDACTED"
                ssl_key => "REDACTED"
        }
}

filter {
        mutate {
                rename => { "_id" => "log_id" }
        }
}

output {
        amazon_es {
                hosts => ["REDACTED"]
                region => "us-east-1"
                index => "REDACTED"
        }
        stdout {}
}

If I run logstash manually with:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d

then it works. There isn't a detected error within auth0.conf.

2

Running /usr/share/logstash/bin/logstash -t --log.level=debug reveals some interesting things ...

Bear in mind that this installation of Logstash has been done via the instructions at Installing Logstash | Logstash Reference [7.9] | Elastic.

  1. The default for path.settings is /usr/share/logstash/config which does not exist. Instead, the installation has put the config into /etc/logstash but hasn't configured Logstash to use that.

  2. If I add --path.settings=/etc/logstash to the test command, I start to get some light into what is causing the ConfigurationError exception ... it is trying to read /etc/logstash/startup.options which isn't a YAML file and, according to the comments in the file "These settings are ONLY used by $LS_HOME/bin/system-install ... and is not used by Logstash itself".

So how do I configure the Logstash service correctly so that it starts and, more importantly, why isn't this being done when installed per the official instructions?

3

Is path.config set in logstash.yml? If so, then it will ignore pipelines.yml. This is actually OK (hence the warning). But when it goes to read the file indicated in logstash.yml, it doesn't find a valid pipeline (hence the error).

4

Yes, it is. OK, so I can ignore the warning ... but I'm still hitting the fact that logstash then tries to read startup.options because it is in /etc/logstash.

At the moment, the only workaround I seem to be able to use is to get logstash to look at /etc/logstash/conf.d instead, so that it ignores everything in /etc/logstash.

It just seems strange to me that the built package isn't working "out of the box" and, as a result, I'm left not knowing what the "right" thing is to do to get it working.

5

Therefore, if path.config is set in logstash.yml, and it's set to something other than auth0.conf, it's not reading auth0.conf...

6

That doesn't make a lot of sense, I'm afraid.

Out of the box, /etc/logstash/pipelines.yml is set to specify path.config as /etc/logstash/conf.d/*.conf, implying (correctly, I hope) that Logstash would then read all of the config files in that directory.

However, out of the box, pipelines.yml is ignored when Logstash is run as a service. This seems to be in part due to the fact that Logstash is configured, by default, to load configuration from /usr/share/logstash/config. One workaround I'm testing is to symlink from there to /etc/logstash and then, in /etc/logstash/logstash.yml, change path.config to /etc/logstash/conf.d in order to avoid the error generated by Logstash trying to parse /etc/logstash/startup.options.

All in all, the default Debian/Ubuntu installation of Logstash is a mess. It does not work and it needs fixing.

7

Do you mind to share what is your OS and how did you install the logstash?

8

It is Ubuntu 18.04.

I followed the instructions in Installing Logstash | Logstash Reference [7.9] | Elastic

9

Just for checking sake, can you share the output of ls -la /etc/logstash

10 
$ ls -la /etc/logstash
total 24
drwxrwxr-x 1 root root  4096 Oct  8 07:48 .
drwxr-xr-x 1 root root  4096 Oct  8 10:04 ..
drwxrwxr-x 1 root root  4096 Sep 23 04:13 conf.d
-rw-r--r-- 1 root root  2019 Sep 23 04:10 jvm.options
-rw-r--r-- 1 root root  9097 Sep 23 04:10 log4j2.properties
-rw-r--r-- 1 root root   342 Sep 23 04:10 logstash-sample.conf
-rw-r--r-- 1 root root 10693 Oct  8 07:48 logstash.yml
-rw-r--r-- 1 root root   285 Sep 23 04:10 pipelines.yml
-rw------- 1 root root  1696 Sep 23 04:10 startup.options
11

Sorry, can you share t he output of ls -laR /etc/logstash once again with recursive?

12

The following is on a clean installation of Logstash:

$ ls -laR /etc/logstash
/etc/logstash:
total 24
drwxrwxr-x 1 root root  4096 Oct  8 07:48 .
drwxr-xr-x 1 root root  4096 Oct  8 10:34 ..
drwxrwxr-x 1 root root  4096 Sep 23 04:13 conf.d
-rw-r--r-- 1 root root  2019 Sep 23 04:10 jvm.options
-rw-r--r-- 1 root root  9097 Sep 23 04:10 log4j2.properties
-rw-r--r-- 1 root root   342 Sep 23 04:10 logstash-sample.conf
-rw-r--r-- 1 root root 10693 Oct  8 07:48 logstash.yml
-rw-r--r-- 1 root root   285 Sep 23 04:10 pipelines.yml
-rw------- 1 root root  1696 Sep 23 04:10 startup.options

/etc/logstash/conf.d:
total 0
drwxrwxr-x 1 root root 4096 Sep 23 04:13 .
drwxrwxr-x 1 root root 4096 Oct  8 07:48 ..

On the system I've been working on, I've got a single .conf file in /etc/logstash/conf.d. I've also been "fiddling" with the various files in /etc/logstash so the datestamps would be newer.

13

I'm trying to do a fresh install of logstash on my ubuntu server to see if I could simulate the same problem. But my environment work as expected and logstash start with no issue.

Below are codes I ran:

java & logstash installation

apt install default-jdk
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt-get update && sudo apt-get install logstash

modify pipelines.yml

vi /etc/logstash/pipelines.yml
- pipeline.id: test
  path.config: "/etc/logstash/conf.d/test.conf"

create test.conf

vi /etc/logstash/conf.d/test.conf
input { stdin { } } output { stdout {} }

Start service

systemctl start logstash
journalctl -xe

Logs

Oct 08 05:15:39 ubuntu logstash[2898]: [2020-10-08T05:15:39,614][INFO ][logstash.javapipeline    ][test] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/test.conf"], :thread=>"#<Thread:0
Oct 08 05:15:40 ubuntu logstash[2898]: [2020-10-08T05:15:40,669][INFO ][logstash.javapipeline    ][test] Pipeline Java execution initialization time {"seconds"=>1.04}
Oct 08 05:15:40 ubuntu logstash[2898]: [2020-10-08T05:15:40,702][INFO ][logstash.javapipeline    ][test] Pipeline started {"pipeline.id"=>"test"}
Oct 08 05:15:40 ubuntu logstash[2898]: [2020-10-08T05:15:40,941][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

I have no idea why you are having this issue too.
Apologize that I couldn't able to help.

14

Actually, you have helped. I tried installing it again on a clean system and, like you, it worked. So I purged it from this system and reinstalled it and now it is working.

So, thank you. I have no idea what has changed. Maybe I fiddled with too many files without remembering :slight_smile:

15

haha.. oh well glad that I could help then!!

16

I'm glad it's working for you. But I believe your problem was that you somehow had path.config set in your logstash.yml file. And this was fixed because you started from scratch with a default set of config files. Of course it's impossible to tell now.

But for any future folks having a similar problem, I'll spell it out:

Logstash reads logstash.yml first. If you have path.config set, you will not be using pipelines.yml. Check logstash.yml first, to see if it's loading a different pipeline than the default.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.