# Hanging shutdown with multiple pipelines. Shutdown of pipelines in wrong order

**URL:** <https://discuss.elastic.co/t/hanging-shutdown-with-multiple-pipelines-shutdown-of-pipelines-in-wrong-order/316354>\
**Category:** Logstash\
**Created:** [October 11, 2022, 3:53pm UTC](https://discuss.elastic.co/t/hanging-shutdown-with-multiple-pipelines-shutdown-of-pipelines-in-wrong-order/316354 "2022-10-11T15:53:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [October 11, 2022, 3:53pm UTC](https://discuss.elastic.co/t/hanging-shutdown-with-multiple-pipelines-shutdown-of-pipelines-in-wrong-order/316354/1 "2022-10-11T15:53:53Z")

</div>

Hi,

I have a logstash config with multiple pipelines. One of the pipeline is a central output which ships to elasticsearch. The other pipelines have their own inputs and filters and having that central output pipeline as output.

I noticed a hanging shutdown because the output pipeline was shut down before the other pipelines. My only solution was to kill the process. I don't want to lose log messages.

Is there a way to configure a pipeline as start first, stop last?  
Currently I am using a memory queue. Would a persistent queue help?

Thanks, Andreas

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [October 11, 2022, 6:05pm UTC](https://discuss.elastic.co/t/hanging-shutdown-with-multiple-pipelines-shutdown-of-pipelines-in-wrong-order/316354/2 "2022-10-11T18:05:35Z")

</div>

Are you using pipeline-to-pipeline communications? If so, the pipeline shutdown calls should be [ordered](https://github.com/elastic/logstash/pull/10872). If you are using (for example) tcp-to-tcp then pipeline shutdown order is not guaranteed.

---

<div class="post-metadata">

**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\
**Post date:** [October 12, 2022, 7:17am UTC](https://discuss.elastic.co/t/hanging-shutdown-with-multiple-pipelines-shutdown-of-pipelines-in-wrong-order/316354/3 "2022-10-12T07:17:29Z")

</div>

Thanks for the reply.  
I am using pipeline to pipeline configuration.

The output of the parsing pipelines looks like the following:

```auto
output
{
	pipeline
	{
		send_to => [commonOut]
 	}
}

```

The pipeline which sends to elasticsearch is using this configuration:

```auto
input
{
  # read from logstash pipeline commonOut
  pipeline
  {
    address => commonOut
  }
}
# encrypted output with TLS and authorization
output
{
	elasticsearch
	{
		hosts => ["${ES_HOSTS}"]
		ssl => "${USE_ES_SSL}"
		cacert => "${ES_CA_CERT_PATH}"
		ssl_certificate_verification	=>	"${USE_ES_OUTPUT_SSL_CERT_VERIFICATION}"

		# credentials are fetched from envrionment or logstash-keystore

		user => "${LOGSTASH_USER}"
		password	=> "${LOGSTASH_PASSWORD}"

		index => "%{[@metadata][indexName]}"
	}
}

```

This is the relevant part of the pipelines.yml:

```auto
# new json format. Contains all logTypes
- pipeline.id: generic_json
  pipeline.workers: 8
  queue.type: memory
  path.config: "/usr/share/logstash/pipelines/generic_json"

- pipeline.id: commonOutElasticsearch
  pipeline.workers: 8
  queue.type: memory
  path.config: "/usr/share/logstash/pipelines/output_es_common_https"

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 24, 2022, 8:11pm UTC](https://discuss.elastic.co/t/hanging-shutdown-with-multiple-pipelines-shutdown-of-pipelines-in-wrong-order/316354/7 "2022-11-24T20:11:22Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.