# Logstash SQS output failing makes pipeline stop working entirely

**URL:** <https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425>\
**Category:** Logstash\
**Created:** [May 6, 2020, 11:26pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425 "2020-05-06T23:26:08Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![robertknutzen](https://avatars.discourse-cdn.com/v4/letter/r/f04885/32.png) [@robertknutzen](https://discuss.elastic.co/u/robertknutzen)\
**Post date:** [May 6, 2020, 11:26pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425/1 "2020-05-06T23:26:08Z")

</div>

I have a logstash pipeline that uses an SQS output. I'm seeing errors where logstash will just stop working if it cannot connect to SQS. The logs seem to indicate that logstash attempts to restart, fails and then just does nothing.

Is there a way to make it try again until it can connect to SQS? It's only happening intermittently and every time the issue is resolved by manually restarting logstash.

Truncated logs below (too big for a post) and [full logs here](https://gist.githubusercontent.com/robertknutzen/9d0ba9726f001e37c5bce4e2bd16c573/raw/771a105429d33ad6b618038fd669b43e5e91d06e/gistfile1.txt)

```auto
[2020-05-06T18:48:35,441][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<Seahorse::Client::NetworkingError: Failed to open TCP connection to sqs.us-west-2.amazonaws.com:443
[2020-05-06T18:48:38,132][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-05-06T18:49:20,643][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"6.4.0"}
[2020-05-06T18:49:47,386][INFO][logstash.pipeline] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>20, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-05-06T18:50:01,463][ERROR][logstash.pipeline] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x615710d4>", :error=>"Failed to open TCP connection t
[2020-05-06T18:50:01,475][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Seahorse::Client::NetworkingError: Failed to open TCP connection to sqs.u
[2020-05-06T18:50:01,512][ERROR][logstash.agent] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAct
[2020-05-06T18:50:01,992][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 7, 2020, 4:25pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425/2 "2020-05-07T16:25:28Z")

</div>

> [@robertknutzen](#):
>
> I'm seeing errors where logstash will just stop working if it cannot connect to SQS.

The plugin [makes one attempt](https://github.com/logstash-plugins/logstash-output-sqs/blob/18f7b8f5c04817d0a42d4c7202c3c89a254cce6d/lib/logstash/outputs/sqs.rb#L90) to connect to SQS during initialization. If that fails it does not retry and will not connect, so nothing happens.

---

<div class="post-metadata">

**Author:** ![robertknutzen](https://avatars.discourse-cdn.com/v4/letter/r/f04885/32.png) [@robertknutzen](https://discuss.elastic.co/u/robertknutzen)\
**Post date:** [May 7, 2020, 5:47pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425/3 "2020-05-07T17:47:07Z")

</div>

Is there anyway to make it try more than once? (other than editing the ruby of the plugin itself obviously)

I'm seeing issues where the machines seem to have a temporary network issue, then just stop working all together. I feel like I've seen different configurations that have an ES output handle this sort of issue gracefully, but I don't have any logs on hand to back up that feeling.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 7, 2020, 5:52pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425/4 "2020-05-07T17:52:20Z")

</div>

> [@robertknutzen](#):
>
> an ES output handle this sort of issue gracefully

Yes, an elasticsearch has code to retry the connection. The sqs input does not.

---

<div class="post-metadata">

**Author:** ![robertknutzen](https://avatars.discourse-cdn.com/v4/letter/r/f04885/32.png) [@robertknutzen](https://discuss.elastic.co/u/robertknutzen)\
**Post date:** [May 7, 2020, 6:04pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425/5 "2020-05-07T18:04:07Z")

</div>

hmm. Ok thanks. Good to know I wasn't just missing the option somewhere.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 4, 2020, 6:04pm UTC](https://discuss.elastic.co/t/logstash-sqs-output-failing-makes-pipeline-stop-working-entirely/231425/6 "2020-06-04T18:04:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.