Logstash Not Starting On Reboot - Ubuntu

Hi
My single node setup works fine except on reboot of the Ubuntu server when Logstash usually does not auto start up. But using the systemctl start logstash works ok.

It looks like the issue is that Logstash is trying to start before the Elasticsearch service is ready on port 9200:

Preformatted text

[2022-11-03T03:30:54,733][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)}
[2022-11-03T03:30:54,733][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}
[2022-11-03T03:30:56,892][ERROR][logstash.outputs.elasticsearch][main][0bdf88fc7228b3da441c9164d1b13821d1db32bc242a6d44e02a97d9fd469f75] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>64}
[2022-11-03T03:30:56,893][ERROR][logstash.outputs.elasticsearch][main][0bdf88fc7228b3da441c9164d1b13821d1db32bc242a6d44e02a97d9fd469f75] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>64}
[2022-11-03T03:30:56,901][ERROR][logstash.outputs.elasticsearch][main][0bdf88fc7228b3da441c9164d1b13821d1db32bc242a6d44e02a97d9fd469f75] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>64}
[2022-11-03T03:30:56,904][ERROR][logstash.outputs.elasticsearch][main][0bdf88fc7228b3da441c9164d1b13821d1db32bc242a6d44e02a97d9fd469f75] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>64}
[2022-11-03T03:30:59,742][INFO ][logstash.outputs.elasticsearch][main] Failed to perform request {:message=>"Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)", :exception=>Manticore::SocketException, :cause=>org.apache.http.conn.HttpHostConnectException: Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)}
[2022-11-03T03:30:59,743][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elastic:xxxxxx@localhost:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://localhost:9200/][Manticore::SocketException] Connect to localhost:9200 [localhost/127.0.0.1] failed: Connection refused (Connection refused)"}

'logstash.service' is unchanged from install:

[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

# When stopping, how long to wait before giving up and sending SIGKILL?
# Keep in mind that SIGKILL on a process can cause data loss.
TimeoutStopSec=infinity

[Install]
WantedBy=multi-user.target

I am not sure if I should somehow delay the startup of log stash until elastic has fully started or would anyone know how to fix the reboot issue?

This my first post and I am new to elastic.

Thanks

Best Regards

I do not know the root cause but if you want to set delay startup of logstash until ES fully started. You can modify Logstash systemd and add Requires to the configuration:
Here is some example that i found o the stackoverflow:

systemd.unit (www.freedesktop.org)
ubuntu 18.04 - Systemd: how to start service after another service started - Server Fault
systemd start a service only if another service runs without errors - Unix & Linux Stack Exchange

tatdat - thank you - I will try Requires

Best Regards

One more thing, If logstash was started before ES, it alway check ES connection continuous until ES is live. Maybe you do not need anything.

The log you shared tells that Elasticsearch is not reachable, this has nothing to do with Logstash starting up or not.

Did you enabled the Logstash service with systemctl enable logstash?

Leandro

Thank you for the response, you are correct, enabling Logstash with systemctl enable logstash (which I must have missed on build) now means that the server reboots and LS runs as expected (without having to start it manually).

Thanks to both contributors.