# If no ls output for a specific data stream does data get discarded?

**URL:** <https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093>\
**Category:** Logstash\
**Created:** [March 22, 2018, 1:52am UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093 "2018-03-22T01:52:46Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ELK\_GB](https://avatars.discourse-cdn.com/v4/letter/e/a9adbd/32.png) [@ELK\_GB](https://discuss.elastic.co/u/ELK_GB)\
**Post date:** [March 22, 2018, 1:52am UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093/1 "2018-03-22T01:52:46Z")

</div>

Hi,

I have a question in rergards to Logstash handling data if no output specified.

For example, I send logs to ES based on some metadata that I specify on the source.

lets say we have the following scenario:

```
output { 
      
    if "log" == [type] and "APACHE" == [product] {
                elasticsearch {
                     hosts => ["es-host1","es-host2"]
                     index => "logs-apache-%{+YYYY.MM.dd}"
                }

     }
    
}

```

So now lets assume that this is it for my output config and I have more products being streamed to this logstash instance, will that data be discarded, if it has no output specified, like the one for apache ?

If it does not go any where then does that cause the data to cause more heap memory usage ?  
As my logstash has a min and max heap size value of 8GB, but sometimes it reaches way more than that up to 11GB and keeps steadily using that much. Is this normal ?

Sorry not an expert on the JVM environment so if there any explanation that would be appreciated.

Thanks

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 22, 2018, 7:09am UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093/2 "2018-03-22T07:09:12Z")

</div>

Once an event gets to the output phase, it is sent to each matching output and then discarded; if there are no matching outputs, then it is discarded without being sent anywhere.

---

<div class="post-metadata">

**Author:** ![ELK\_GB](https://avatars.discourse-cdn.com/v4/letter/e/a9adbd/32.png) [@ELK\_GB](https://discuss.elastic.co/u/ELK_GB)\
**Post date:** [March 22, 2018, 12:43pm UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093/3 "2018-03-22T12:43:58Z")

</div>

Great, thanks for the answer !

I was wondering as I'm comparing few other tools and the behavior of such scenario.

Any idea about the heap usage in logstash being way over the limit ?

Thanks

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 22, 2018, 8:24pm UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093/4 "2018-03-22T20:24:43Z")

</div>

> [@ELK\_GB](#):
>
> As my logstash has a min and max heap size value of 8GB, but sometimes it reaches way more than that up to 11GB

It sounds like your settings aren't getting respected; how are you configuring your heap settings, and how is Logstash being run (e.g., via command line, or as a OS-managed service using something like `upstart` or `initctl`)?

---

<div class="post-metadata">

**Author:** ![ELK\_GB](https://avatars.discourse-cdn.com/v4/letter/e/a9adbd/32.png) [@ELK\_GB](https://discuss.elastic.co/u/ELK_GB)\
**Post date:** [March 23, 2018, 6:06pm UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093/5 "2018-03-23T18:06:43Z")

</div>

> [@yaauie](#):
>
> how are you configuring your heap settings

I edit he `jvm.options` file located under /etc/logstash/

My understanding is that logstash shouldn't go much beyond my min and max, meaning why I see logstash jumping up to almost 15GB of RAM usage, which is almost equivalent to my machine total RAM, when I have the heap set at 8GB. Is that Normal behavior ?

Under great pressure losgtash will never be less than 10GB RAM usage, most of the time.

> [@yaauie](#):
>
> how is Logstash being run (e.g., via command line, or as a OS-managed service using something like upstart or initctl)?

I use os managed service in my case initctl

My concern here is why the memory usage is high. When monitoring my OS resources the java process corespsong to Logstash is always reping +10GB for logstash.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 20, 2018, 6:07pm UTC](https://discuss.elastic.co/t/if-no-ls-output-for-a-specific-data-stream-does-data-get-discarded/125093/6 "2018-04-20T18:07:03Z")

</div>

Closed on April 20, 2018, 6:07pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.