# Retry on failure mechanism in logstash-bigquery filter plugin

**URL:** <https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719>\
**Category:** Logstash\
**Created:** [August 28, 2024, 5:50pm UTC](https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719 "2024-08-28T17:50:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Deena\_Dayalan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deena_dayalan/32/135951_2.png) [@Deena\_Dayalan](https://discuss.elastic.co/u/Deena_Dayalan)\
**Post date:** [August 28, 2024, 5:50pm UTC](https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719/1 "2024-08-28T17:50:04Z")

</div>

Recently, I encountered an issue where events uploads to BigQuery from Logstash failed with a `BigQueryException: The service is currently unavailable`  
error. My concern is that valid events might fail due to BigQuery service unavailability. In this scenario, does Logstash not automatically retry sending the failed events? While I understand that these events are stored in the `error_directory`, I have noticed that the Logstash-BigQuery pipeline lacks a built-in retry mechanism. Is there any way to implement a retry mechanism within the BigQuery filter for failed events? Does Logstash have any internal retry mechanism for handling such failures?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 28, 2024, 6:18pm UTC](https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719/2 "2024-08-28T18:18:18Z")

</div>

The documentation says you can "manually fix the events" and then (presumably) re-route them back through the pipeline. logstash is not going to handle that internally.

---

<div class="post-metadata">

**Author:** ![ashishtiwari1993](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ashishtiwari1993/32/135241_2.png) [@ashishtiwari1993](https://discuss.elastic.co/u/ashishtiwari1993)\
**Post date:** [August 29, 2024, 4:15am UTC](https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719/3 "2024-08-29T04:15:09Z")

</div>

Hi @Deena_Dayalan, You can use [Dead letter queues](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html), The dead letter queue (DLQ) is designed as a place to temporarily write events that cannot be processed. Later you can [process](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html#processing-dlq-events) all the events from DLQ.

---

<div class="post-metadata">

**Author:** ![Deena\_Dayalan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deena_dayalan/32/135951_2.png) [@Deena\_Dayalan](https://discuss.elastic.co/u/Deena_Dayalan)\
**Post date:** [August 29, 2024, 7:39am UTC](https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719/4 "2024-08-29T07:39:08Z")

</div>

Thanks @Badger @ashishtiwari1993. I see the setting "error\_directory" itself act as kinda DLQ in BQ output plugin.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 29, 2024, 12:45pm UTC](https://discuss.elastic.co/t/retry-on-failure-mechanism-in-logstash-bigquery-filter-plugin/365719/5 "2024-08-29T12:45:35Z")

</div>

> [@ashishtiwari1993](#):
>
> You can use [Dead letter queues](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html)

No, you cannot. The elasticsearch output uses a DLQ because the code [invokes the dlq writer](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/c0525077319c43eb36d0fc87f6366d9af3afc5f2/lib/logstash/outputs/elasticsearch.rb#L405). (It may be the only output plugin that does so.) The equivalent code in the bigquery output [writes to the error directory](https://github.com/logstash-plugins/logstash-output-google_bigquery/blob/c29ff8df877fac02ac61282de8fb6d9eaab5e112/lib/logstash/outputs/google_bigquery.rb#L260).

That method in the bigquery output could be modified to use a DLQ, but it does not do so as-is.