Output to multiple sources is causing missing logs

Elastic Stack Logstash
1

I have a conf file that inputs different files and assigns them a [type] so that they can be accessed individually with the filter and output sections of the conf file.

Here is my conf file:

As you can see, I am sending the [type] = "bluecoat_proxy" to multiple outputs using different plugins.

My issue is that when I was only sending to elasticsearch, all of the logs were being delivered properly. Now that I am using several output plugins, some of the logs are going to elasticsearch, some are going to syslog, and some are going to tcp. To make this situation even odder is that this does not happen every time. These logs come in as a .log file and contain 1 hour of Symantec (Bluecoat) WSS logs. The size of this log file can range from 50 MB to several hundred MB.

I was wondering if there is something with the file queue or processing of events that may need to change in order to fix this or if there is something I am not doing properly with my output section of my conf file.

Thank you for your help

2

With that configuration I would expect every event to go to all three outputs (udp can silently lose data, but the tcp and elasticsearch outputs should get everything). Are there any warnings in the logstash log?

3

Thank you for looking at this. I am not exactly sure why it did this during a 36 hour stretch but has not done so since I made the original post. All I did was a typical 'IT Remdiation': reboot the system.

Thank you again

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.