Event loss on way rsyslog-> LS -> ES

Elastic Stack Elasticsearch
21

Ouput of /_cat/indices

yellow open proxy-test 5 1 11087461 0 10.3gb 10.3gb

Output of # cat test.log | wc -l
11087461

It's exactlly the same... I've configured the matching logstash filter as follows:

input

   file {
    path => "test.log"
    start_position => "beginning"
    type => "test"
  }

filter
else if [type] == "test" {
And without if [program] == "accesslog-elastic" {

What's wrong with this condition: if [program] == "accesslog-elastic" ?

Thank you

22

Okay, it seems like rsyslog was overhelmed by reverse dns resolving.

I've started rsyslog with option -x

rsyslogd -x
23

new metric:

#events.rate_15m
 3,176.769
#events.rate_1m
 3,121.133 
#events.rate_5m
 3,161.205

I'll do some another validations to make sure that all event are recieved

24

Now, it seems like logstash itself is the bottleneck... Logs of proxy are nearly correct. But ASA Logs not

netstat -c --udp -an
Proto Recv-Q Send-Q Local Address               Foreign Address             State                              
udp   121936      0 :::10514                    :::*

Maybe to much events? Is there any opportunity to increase size of the queue or building a qeueu system ?