# Is Logstash executing config sequentially

**URL:** <https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642>\
**Category:** Logstash\
**Created:** [August 30, 2018, 8:24am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642 "2018-08-30T08:24:03Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [August 30, 2018, 8:24am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/1 "2018-08-30T08:24:03Z")

</div>

Hi there.  
I have two Elasticsearch clusters so Logstash has two output definitions for every index in the way like this:

```
if [type] == "iis"
{
	elasticsearch
	{
	hosts => ["NODE1:9200", "NODE2:9200"]
		index => "logstash-%{[type]}-%{+YYYY.MM}" 
		template_overwrite => true
	}
	elasticsearch
	{
	hosts => ["A-NODE1:9200", "A-NODE2:9200"]
		index => "logstash-%{[type]}-%{+YYYY.MM}"
		template_overwrite => true
	}

```

}

One node from first cluster run out of space so all indices became readonly.  
I thought that second cluster should get logs, but it's not.

Is Logstash executing config sequentially, like source code? Is it correct that if any exception occur then all subsequent steps won't be executed?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 30, 2018, 8:55am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/2 "2018-08-30T08:55:30Z")

</div>

Logstash pushes data to outputs synchronously so a problem with one output will affect all outputs.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [August 30, 2018, 9:02am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/3 "2018-08-30T09:02:48Z")

</div>

May be there is a workaround except of deploying second logstash?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 30, 2018, 9:04am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/4 "2018-08-30T09:04:31Z")

</div>

I don't think there is.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [September 13, 2018, 10:54am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/5 "2018-09-13T10:54:08Z")

</div>

Another one question related to this problem?  
Where does log go when beats or nxlog sent it to logstash but the last one can't restranslate it to ES?  
Does logstash have some kind of buffer for this case?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 13, 2018, 11:02am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/6 "2018-09-13T11:02:26Z")

</div>

Yes, if you enable the persistent queue.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [September 13, 2018, 11:10am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/7 "2018-09-13T11:10:39Z")

</div>

But what could happen if persistent queues is not enabled?  
Event will store in-memory until what?  
If logstash service restarts, all queued events will be lost?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 13, 2018, 12:25pm UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/8 "2018-09-13T12:25:03Z")

</div>

> But what could happen if persistent queues is not enabled?  
> Event will store in-memory until what?

Don't count on the in-memory store. It's _very_ small. If it gets full Logstash will stop accepting new events which hopefully makes the sending party stop pushing more data. I don't know how NXLog behaves in this case, but Filebeat will simple let the events rest in the log files, i.e. the original log files become the buffer.

> If logstash service restarts, all queued events will be lost?

Yes.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [September 13, 2018, 12:28pm UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/9 "2018-09-13T12:28:39Z")

</div>

> [@magnusbaeck](#):
>
> > If logstash service restarts, all queued events will be lost?
> 
> Yes.

Even if shutdown was safe like written here?[Shutting Down Logstash | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/shutdown.html)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 13, 2018, 12:56pm UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/10 "2018-09-13T12:56:32Z")

</div>

You're right, if the shutdown is graceful _and_ Logstash is able to wait for the internal queue to drain nothing will be lost. But, if the queue is full because the outputs are clogged Logstash can't do anything unless you've enabled the persistent queue.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [September 13, 2018, 1:04pm UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/11 "2018-09-13T13:04:08Z")

</div>

Thank you for your answers

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [September 21, 2018, 2:41pm UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/12 "2018-09-21T14:41:30Z")

</div>

I have another suggestion to check that is related to this thread.

Let's assume that logstash has two ES outputs. Connection with one of them is unstable.  
Am I understand right that logstash will push events to BOTH ES synchroneously? So if one of outputs has unstable network connection then second output with good connection will not recieve events from logstash too. Is this correct?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 21, 2018, 4:02pm UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/13 "2018-09-21T16:02:21Z")

</div>

Yes.

---

<div class="post-metadata">

**Author:** ![74db36a597f21b891b3f](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/74db36a597f21b891b3f/32/45367_2.png) [@74db36a597f21b891b3f](https://discuss.elastic.co/u/74db36a597f21b891b3f)\
**Post date:** [September 22, 2018, 11:11am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/14 "2018-09-22T11:11:07Z")

</div>

Is there any way to debug such behaviour, when events are stuck in logstash queue? Which logger responds for that?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2018, 11:11am UTC](https://discuss.elastic.co/t/is-logstash-executing-config-sequentially/146642/15 "2018-10-20T11:11:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.