If no ls output for a specific data stream does data get discarded?

Elastic Stack Logstash
1

Hi,

I have a question in rergards to Logstash handling data if no output specified.

For example, I send logs to ES based on some metadata that I specify on the source.

lets say we have the following scenario:

output { 
      
    if "log" == [type] and "APACHE" == [product] {
                elasticsearch {
                     hosts => ["es-host1","es-host2"]
                     index => "logs-apache-%{+YYYY.MM.dd}"
                }

     }
    
}

So now lets assume that this is it for my output config and I have more products being streamed to this logstash instance, will that data be discarded, if it has no output specified, like the one for apache ?

If it does not go any where then does that cause the data to cause more heap memory usage ?
As my logstash has a min and max heap size value of 8GB, but sometimes it reaches way more than that up to 11GB and keeps steadily using that much. Is this normal ?

Sorry not an expert on the JVM environment so if there any explanation that would be appreciated.

Thanks

2

Once an event gets to the output phase, it is sent to each matching output and then discarded; if there are no matching outputs, then it is discarded without being sent anywhere.

3

Great, thanks for the answer !

I was wondering as I'm comparing few other tools and the behavior of such scenario.

Any idea about the heap usage in logstash being way over the limit ?

Thanks

4

It sounds like your settings aren't getting respected; how are you configuring your heap settings, and how is Logstash being run (e.g., via command line, or as a OS-managed service using something like upstart or initctl)?

5

I edit he jvm.options file located under /etc/logstash/

My understanding is that logstash shouldn't go much beyond my min and max, meaning why I see logstash jumping up to almost 15GB of RAM usage, which is almost equivalent to my machine total RAM, when I have the heap set at 8GB. Is that Normal behavior ?

Under great pressure losgtash will never be less than 10GB RAM usage, most of the time.

I use os managed service in my case initctl

My concern here is why the memory usage is high. When monitoring my OS resources the java process corespsong to Logstash is always reping +10GB for logstash.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.