Systemctl start logstash does nothing

[EvanGertis]

I am working on pulling changes onto a few logstash boxes. The issue is when I run the command

systemctl start logstash

I get nothing back.

Expected:
logstash starts with a verbose output.

Actual:

Any help with this issue would be greatly appreciated. Thank you.

[Badger]

@EvanGertis

I get nothing back.

That is normal. You can check the systemd journal using

journalctl -u logstash

[EvanGertis]

I get back

journalctl -u logstash
-- Logs begin at Fri 2021-11-05 19:37:17 UTC, end at Thu 2021-12-02 19:16:19 UTC. --
Nov 30 13:51:52 ip-10-12-128-146.ec2.internal logstash[16366]: [2021-11-30T13:51:52,167][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"zitilogs-2021>
Nov 30 13:51:52 ip-10-12-128-146.ec2.internal logstash[16366]: [2021-11-30T13:51:52,167][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"zitilogs-2021>

I'd like to make sure that the changes that I've have taken place.

[EvanGertis]

The logs also stopped flowing to elastic.

[Badger]

The other place to check is /var/log/logstash.

[EvanGertis]

I'm getting

[FATAL] 2021-12-02 19:24:16.596 [LogStash::Runner] runner - The given configuration is invalid. Reason: Cannot evaluate `${LOGSTASH_PASSWORD}`. Replacement variable `LOGSTASH_PASSWORD` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.

[EvanGertis]

I've updated the env vars and I ran

bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2021-12-02 19:34:51.986 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-12-02 19:35:17.674 [LogStash::Runner] cloudwatchlogs - Using version 0.1.x codec plugin 'cloudwatch_logs'. This plugin isn't well supported by the community and likely has no maintainer.
Configuration OK

But now I get

# systemctl status logstash
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: deactivating (stop-sigterm) since Thu 2021-12-02 16:15:29 UTC; 3h 22min ago
 Main PID: 16366 (java)
    Tasks: 35 (limit: 22600)
   Memory: 1.6G
   CGroup: /system.slice/logstash.service

I was able to get through this by finally running

bin/logstash -f /etc/logstash/conf.d/
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2021-12-02 19:54:46.708 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2021-12-02 19:54:46.726 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.8.20"}