Logstash stops inputs after update

Elastic Stack Logstash
1

We currently try to update our logstash servers from 8.2.0 to 8.18.x (and later 9.x).
In Version 8.18.x we face an issue with logstash. I tried to find the exact version which is failing and found that with 8.11.4 everything runs as expected while 8.12.0 seems to introduce some changes I oversee.

Now here comes the problem. Logstash seems to be running (systemd service) but alle the pipelines are stopped. With the debug-log enabled I see that logstash stops all inputs directly after they were started.

[2025-07-03T12:41:42,464][INFO ][logstash.agent           ] Pipelines running {:count=>3, :running_pipelines=>[:".monitoring-logstash", :some_pipeline, :my_input], :non_running_pipelines=>[]}
[2025-07-03T12:41:42,468][DEBUG][io.netty.util.internal.PlatformDependent][my_input][input_beats_ssl] org.jctools-core.MpscChunkedArrayQueue: available
[2025-07-03T12:41:42,467][DEBUG][logstash.agent           ][some_pipeline] Executing action {:action=>LogStash::PipelineAction::Stop/pipeline_id:some_pipeline}
[2025-07-03T12:41:42,468][DEBUG][logstash.javapipeline    ][some_pipeline] Closing inputs {:pipeline_id=>"some_pipeline", :thread=>"#<Thread:0x769f6470 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2025-07-03T12:41:42,468][DEBUG][logstash.plugins.builtin.pipeline.input][some_pipeline] Stopping {:plugin=>"LogStash::Plugins::Builtin::Pipeline::Input"}
[2025-07-03T12:41:42,473][DEBUG][logstash.javapipeline    ][some_pipeline] Closed inputs {:pipeline_id=>"some_pipeline", :thread=>"#<Thread:0x769f6470 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}
[2025-07-03T12:41:42,475][INFO ][org.logstash.beats.Server][my_input][input_beats_ssl] Starting server on port: 5055
[2025-07-03T12:41:42,483][INFO ][org.logstash.beats.Server][my_input][input_beats_default] Starting server on port: 5044
[2025-07-03T12:41:42,495][DEBUG][io.netty.channel.DefaultChannelId][my_input][input_beats_default] -Dio.netty.processId: 75557 (auto-detected)
[2025-07-03T12:41:42,497][DEBUG][io.netty.util.NetUtil    ][my_input][input_beats_default] -Djava.net.preferIPv4Stack: false
[2025-07-03T12:41:42,497][DEBUG][io.netty.util.NetUtil    ][my_input][input_beats_default] -Djava.net.preferIPv6Addresses: false
[2025-07-03T12:41:42,499][DEBUG][io.netty.util.NetUtilInitializations][my_input][input_beats_default] Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
[2025-07-03T12:41:42,500][DEBUG][io.netty.util.NetUtil    ][my_input][input_beats_default] /proc/sys/net/core/somaxconn: 4096
[2025-07-03T12:41:42,501][DEBUG][io.netty.channel.DefaultChannelId][my_input][input_beats_default] -Dio.netty.machineId: 00:50:56:ff:fe:af:97:1b (auto-detected)
[2025-07-03T12:41:42,507][DEBUG][logstash.plugins.builtin.pipeline.input][some_pipeline][2002e3502270c1735eb64a4ffdd65e198dab5bad45e95ca50304ed62171fb0ca] Closing {:plugin=>"LogStash::Plugins::Builtin::Pipeline::Input"}
[2025-07-03T12:41:42,508][DEBUG][logstash.pluginmetadata  ][some_pipeline][2002e3502270c1735eb64a4ffdd65e198dab5bad45e95ca50304ed62171fb0ca] Removing metadata for plugin 2002e3502270c1735eb64a4ffdd65e198dab5bad45e95ca50304ed62171fb0ca
[2025-07-03T12:41:42,509][DEBUG][logstash.javapipeline    ][some_pipeline] Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>"some_pipeline", :thread=>"#<Thread:0x769f6470 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2025-07-03T12:41:42,510][DEBUG][logstash.javapipeline    ][some_pipeline] Shutdown waiting for worker thread {:pipeline_id=>"some_pipeline", :thread=>"#<LogStash::WorkerLoopThread:0x6aa3c0f0 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300 sleep>"}
[2025-07-03T12:41:42,565][DEBUG][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash][65d01777d62b982ef0a2d1a7dbb4065d54a95123b05bc1d78aa2397ae1b7538e] Sending final bulk request for batch. {:action_count=>2, :payload_size=>91786, :content_length=>24352, :batch_offset=>0}
[2025-07-03T12:41:42,581][INFO ][logstash.outputs.elasticsearch][some_pipeline][d568ccb36ccd9cc978a282ebc0582621de0000052f29645dea225068347203fc] Aborting the batch due to shutdown request while waiting for connections to become live
[2025-07-03T12:41:42,586][INFO ][org.logstash.execution.WorkerLoop][some_pipeline] Received signal to abort processing current batch. Terminating pipeline worker [some_pipeline]>worker3
[2025-07-03T12:41:42,684][INFO ][logstash.outputs.elasticsearch][some_pipeline][d568ccb36ccd9cc978a282ebc0582621de0000052f29645dea225068347203fc] Aborting the batch due to shutdown request while waiting for connections to become live
[2025-07-03T12:41:42,686][INFO ][org.logstash.execution.WorkerLoop][some_pipeline] Received signal to abort processing current batch. Terminating pipeline worker [some_pipeline]>worker2
[2025-07-03T12:41:42,844][INFO ][logstash.outputs.elasticsearch][some_pipeline][d568ccb36ccd9cc978a282ebc0582621de0000052f29645dea225068347203fc] Aborting the batch due to shutdown request while waiting for connections to become live
[2025-07-03T12:41:42,847][INFO ][org.logstash.execution.WorkerLoop][some_pipeline] Received signal to abort processing current batch. Terminating pipeline worker [some_pipeline]>worker1

Is there somebody who knows what is going on? I am happy for any help.

2

You need to share your pipelines.

3

Thank you leandrojmp!

Obviously the pipeline-configuration is crucial. I found the part that is causing the problems but still did not get it to work. Problem is the ssl-part of the elasticsearch-output.
Because the complete config is very long and I cannot post it here I tried to reduce it to the important parts. Let me know if something is missing.

This is my original config

output {
    elasticsearch {
      user => "es-user"
      password => "secret"
      ssl => true
      ssl_certificate_verification => true
      cacert => "/etc/ssl/custom/ca.crt"
      hosts => "my-elastic1.some.domain:9200"
      hosts => "my-elastic2.some.domain:9200"
      index => "%{[@metadata][index]}"
    }
}

I know it is causing the problems as this one works: (by works I mean the pipelines do not get killed)

output {
    elasticsearch {
      user => "es-user"
      password => "secret"
      ssl => false
      #ssl_certificate_verification => true
      #cacert => "/etc/ssl/custom/ca.crt"
      hosts => "my-elastic1.some.domain:9200"
      hosts => "my-elastic2.some.domain:9200"
      index => "%{[@metadata][index]}"
    }
}

Now for what I tried as a solution. I have overseen the deprecation-warnings for the ssl-settings. They got deprecated with v11.14.0 of the elasticsearch-output-plugin. I am trying to use the correct settings

output {
    elasticsearch {
      user => "es-user"
      password => "secret"
      ssl_enabled => true
      ssl_verification_mode => full
      ssl_certificate_authorities => ["/etc/ssl/custom/ca.crt"]
      hosts => "my-elastic1.some.domain:9200"
      hosts => "my-elastic2.some.domain:9200"
      index => "%{[@metadata][index]}"
    }
}

But I have the same problems. The pipelines still get shut down.

[2025-07-04T07:38:17,871][ERROR][logstash.outputs.elasticsearch][some_pipeline] Failed to bootstrap. Pipeline "some_pipeline" is going to shut down {:message=>"Got response code '403' contacting Elasticsearch at URL 'https://my-elastic1.some.domain/_index_template/logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError}

This log line is introduced here shut down pipeline when finish register fails by kaisecheng · Pull Request #1151 · logstash-plugins/logstash-output-elasticsearch · GitHub. Still I do not know how to fix this.

4

Figured it out by myself. The last logline actually had all information required.
We use readonlyrest to control access to different logs and the elasticsearch-output requires more actions now.

I know this forum is not for readonlyrest, I did not expect this to be caused by readonlyrest plugin. Posting the solution here for completeness.

In this case logstash wanted to find (or create) an index-template named "logstash". So I needed to allow two more actions:

        - "indices:admin/index_template/get"
        - "indices:admin/index_template/put"
1 Like