Hello,
we are running logstash 8.5.0 with multiple pipelines outputting to elasticsearch.
Most of the time this works fine, but sometimes logstash will ignore the datastream configuration on startup and tries to write into the ecs-logstash index.
We cannot reproduce this behaviour consistently, but it will happen every 5 restarts or so.
Logstash will then continously write the following error message:
[2022-10-20T11:12:23,719][ERROR][logstash.outputs.elasticsearch][logs-fsecure-prod][a0f441f48a90bca64a9e011a7f67a0e99c01a321f9792cbf83dc7ed9e81f80f8] Elasticsearch setup did not complete normally, please review previously logged errors {
:message=>"Got response code '403' contacting Elasticsearch at URL 'https://elastic01p.XXXXX:9200/ecs-logstash'", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError}
There are several posts about this issue:
- Logstash sometimes ignoring datastream configuration in elasticsearch output
- Possible race condition causing Elasticsearch output plugin to overwrite supplied index with rollover alias
But in our case we see that error for pipelines where we have several outputs to different elasticsearches in one pipeline and some use ilm/index and some use datastreams. Seems like errors and problems caused by logstash logic which treats the whole output like one output and not like several different outputs for different elasticsearch clusters.
In debug mode we see errors that datastream can not be used with ilm but as I stated above we do not combine ilm and datastream options and use ilm for one elasticsearch cluster and datastream for another elasticsearch cluster but inside one output. That is why we think that check config logic is wrong.
Why stop/start solves problem with output to ecs-logstash instead of datastream is not clear and maybe it's connected with wrong logic described above.