# Logstash retries 404 instead of dropping - output stuck

**URL:** <https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741>\
**Category:** Logstash\
**Created:** [November 24, 2022, 11:08am UTC](https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741 "2022-11-24T11:08:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Harm](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Harm](https://discuss.elastic.co/u/Harm)\
**Post date:** [November 24, 2022, 11:08am UTC](https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741/1 "2022-11-24T11:08:20Z")

</div>

Hi!

Today I had a situation where one of our Elasticsearch outputs in logstash got unavailable. It replied with a 404. Based upon the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html) logstash should give a warning and drop the event. We have no DLQ configured.

> 400 and 404 errors are sent to the dead letter queue (DLQ), if enabled. If a DLQ is not enabled, a log message will be emitted, and the event will be dropped. See [DLQ Policy](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-dlq-policy) for more info.

However since it got stuck retrying, it seemed like all other outputs were also not fowarding data anymore.

Output: `[ERROR][logstash.outputs.elasticsearch][main] Encountered a retryable error (will retry with exponential backoff) {:code=>404, :url=>"https://our-url/_bulk", :content_length=>22, :body=>"{\"ok\":false,\"message\":\"Unknown resource.\"}\n"}`

Logstash version: 7.17.6.1

Configuration:

```auto
    elasticsearch {
      hosts => ["https://our-url:443"]
      index => "%{[index][name]}"
      ilm_enabled => false
      manage_template => false
      action => "create"
      ecs_compatibility => "v8"
      user => "logstash_writer"
      password => " **********"
      failure_type_logging_whitelist => ["version_conflict_engine_exception"]
    }

```

Is this a bug? Or have i missed something?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 24, 2022, 5:31pm UTC](https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741/2 "2022-11-24T17:31:17Z")

</div>

> [@Harm](#):
>
> Or have i missed something?

You missed something 🙂 Before reading the documentation for the [Retry Policy](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_retry_policy) you need to understand what a \_bulk request to elasticsearch looks like. It is an HTTP request that contains indexing requests for multiple documents. The overall request gets an HTTP status back. If that is not 200 then the documentation says it is retried indefinitely. The individual documents in the \_bulk request also get HTTP statuses back. If the status for a document is 400 or 404 then it is routed to the DLQ.

The bulk request and DLQ logic is [here](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/91a5ecda38862d882ecd3337107a3ae937adc585/lib/logstash/plugin_mixins/elasticsearch/common.rb#L340).

---

<div class="post-metadata">

**Author:** ![Harm](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Harm](https://discuss.elastic.co/u/Harm)\
**Post date:** [November 24, 2022, 6:29pm UTC](https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741/3 "2022-11-24T18:29:27Z")

</div>

> [@Badger](#):
>
> You missed something 🙂

Hehe, thanks. That makes at lot of sense. Do you have any suggestion to not let such issues bring down a logstash instance completly? I guess my main issue is that there are different outputs in the same pipeline?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 24, 2022, 7:08pm UTC](https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741/4 "2022-11-24T19:08:08Z")

</div>

You can cope with a limited outage of one output by using pipeline to pipeline communication with an [output isolator](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#output-isolator-pattern) pattern.

If you want to drop data instead of using logstash's at-least-once delivery model then see [this post](https://discuss.elastic.co/t/logstash-multiple-pipelines-failure-with-persistent-queues/296520/2).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 22, 2022, 7:08pm UTC](https://discuss.elastic.co/t/logstash-retries-404-instead-of-dropping-output-stuck/319741/5 "2022-12-22T19:08:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.