Logstash Gone Wrong After force shutdown

[akrog79]

I have logstash installed as a service on a machine with Logstash+Kibana+Elastic.

My logstash was updated, so its seems that doesn't shutdown properly and now show an error with a pipeline:

Jan 26 10:19:56 esearch logstash[29197]: [2024-01-26T10:19:56,748][INFO ][logstash.outputs.elasticsearch][pipeline][dec4e13358daa92f56dcbdf7685e6d1c745c7be9f58d718cb6dda6176d841907] Aborting the batch due to shutdown request while waiting for connections to become live
Jan 26 10:19:56 esearch logstash[29197]: [2024-01-26T10:19:56,758][INFO ][org.logstash.execution.WorkerLoop][pipeline] Received signal to abort processing current batch. Terminating pipeline worker [pipeline]>worker5

I only want that pipeline start and ingest data. How can I solve it?

[leandrojmp (Leandro Pereira)]

Hello,

There are no errors in the logs that you shared, both are INFO logs, it is not clear what is the issue.

Have you tried to restart your logstash service?

[akrog79]

Yes, I tried to restart logstash service but doesn't solve the problem, returns the same log.

The log show that the pipeline is terminated, this is an error, no?

[leandrojmp (Leandro Pereira)]

@akrog79

The log show that the pipeline is terminated, this is an error, no?

No, errors are logged as ERROR, this is an INFO log.

This is a message saying that a shutdown for your logstash was requested.

What was the version you are using and what is the updated version? How you updated it?

Also, what else do you have in the logs?

Have you tried to stop the service and then start again, not just a restart command?

[akrog79]

Yes, I tried to stop and start service, made a complete machine reboot...

I upgrade to 8.12 from 8.11 via zypper update.

How can I check more logs? logstash-plain.log show the same logs that I share with you.

[Rios]

Check this topic, maaaaybe data has been locked.

I'm suggest to check ES, reason: [logstash.outputs.elasticsearch][pipeline][dec4e13358daa92f56dcbdf7685e6d1c745c7be9f58d718cb6dda6176d841907] Aborting the batch due to shutdown request while waiting for connections to become live

[akrog79]

not work deleting .lock file

[Rios]

Have you restarted LS the service?

Have you check ES, does it work from LS host?

[akrog79]

No... Can you explain me how I should do it?

[Rios]

LS: systemcl restart logstash

ES:

1. Test by curl:
   curl -u user:pass -v https://eshost:9200/_cat/health?v=true

2. Check ES log " /var/log/elasticsearch/"

[Ritikapawar]

If systemctl restart logstash is not working you might need to kill the whole process follow the document Shutting Down Logstash | Logstash Reference [8.12] | Elastic.
and also check the status of elasticsearch

[Rios]

If he had restarted LS, should be normally terminated. However it's good to check what Ritikapawar suggested.
systemctl stop logstash
ps aux | grep logstash

[akrog79]

I see a new type of error

Failed to install template {:message=>"Failed to load template file '/etc/logstash/elastic-fortigate-template.json': Unrecognized token 'te': was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')\n at [Source: (byte[])\"te\": \"fortigate*\",\n \"settings\": {\n \"index.refresh_interval\": \"5s\",\n \"number_of_shards\" : 1,\n \"number_of_replicas\" : 0\n },\n \"mappings\": {\n \"fortigate\": {\n \"dynamic_templates\": [\n {\n \"notanalyzed\": {\n \"match\": \"*\",\n \"match_mapping_type\": \"string\",\n \"mapping\": {\n \"type\": \"string\",\n \"index\": \"not_analyzed\",\n \"doc_values\": \"true\"\n }\n }\n }\n ],\n \"properties\": \"[truncated 7532 bytes]; line: 1, column: 4]", :exception=>LogStash::ConfigurationError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:106:inread_template_file'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:23:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/outputs/elasticsearch.rb:663:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/outputs/elasticsearch.rb:371:in finish_register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/outputs/elasticsearch.rb:328:in block in register'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-elasticsearch-11.22.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:172:in `block in after_successful_connection'"]}