Logstash sometimes ignoring datastream configuration in elasticsearch output

Elastic Stack Logstash
1

Hello,
we are running logstash 8.5.0 with multiple pipelines outputting to elasticsearch.
Most of the time this works fine, but sometimes logstash will ignore the datastream configuration on startup and tries to write into the ecs-logstash index.
We cannot reproduce this behaviour consistently, but it will happen every 5 restarts or so.

Logstash will then continously write the following error message:

[2022-10-20T11:12:23,719][ERROR][logstash.outputs.elasticsearch][logs-fsecure-prod][a0f441f48a90bca64a9e011a7f67a0e99c01a321f9792cbf83dc7ed9e81f80f8] Elasticsearch setup did not complete normally, please review previously logged errors {
:message=>"Got response code '403' contacting Elasticsearch at URL 'https://elastic01p.XXXXX:9200/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError}

There are several posts about this issue:

But in our case we see that error for pipelines where we have several outputs to different elasticsearches in one pipeline and some use ilm/index and some use datastreams. Seems like errors and problems caused by logstash logic which treats the whole output like one output and not like several different outputs for different elasticsearch clusters.

In debug mode we see errors that datastream can not be used with ilm but as I stated above we do not combine ilm and datastream options and use ilm for one elasticsearch cluster and datastream for another elasticsearch cluster but inside one output. That is why we think that check config logic is wrong.

Why stop/start solves problem with output to ecs-logstash instead of datastream is not clear and maybe it's connected with wrong logic described above.

2

Did you set datastream => true/false explicitly in all output configurations?

Because in Logstash 8 it decides automatically if it uses the datastream config or not, based on the existing fields on the event that is to be written to Elasticsearch (See Elasticsearch output plugin | Logstash Reference [8.5] | Elastic). This could maybe happen due to different events that are going through these ouput.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.