@steffens @ruflin Spoke too soon. Logstash just isn't generating any messages that it is dropping events when I use the beats plugin.
From the filebeat debug log:
$ grep "source" filebeat | grep -v sourcename | wc -l
7974
That is the correct number of events that it should send.
Each event will have a single DOY field so I enabled debugging on logstash and ran the files through again after deleting the C:\ProgramData\filebeat folder:
$ grep "source" filebeat | grep -v sourcename | wc -l
7974
$ grep '"DOY" =>' logstash.stdout | wc -l
484
That's missing 7490 events...
Only six events are recorded in /var/log/logstash/logstash.log and they are all similar to this.
{:timestamp=>"2015-11-06T14:49:46.442000-0600", :message=>"Failed parsing date from field", :field=>"Date", :value=>"2015-10-05 %{Hour}:%{Minute}", :exception=>"Invalid format: "2015-10-05 %{Hour}:%{Minute}"", :config_parsers=>"yyyy-MM-dd HH:mm", :config_locale=>"default=en_US", :level=>:warn}
This is expected since it is because the file is a csv and the first line doesn't match the filter. There should be far more than 6 events like this.
After this I cleared the the logs and ES:
# echo $null > /var/log/logstash.log
echo $null > /var/log/logstash.stdout
With logstash-forwarder sending the events I checked logstash.log and see:
{:timestamp=>"2015-11-06T15:00:49.869000-0600", :message=>"too many attempts at sending event. dropping: 2015-10-20T04:57:00.000Z HLCEAVM 23:57,7,0,6,20,24,6276,6760,19,21,1803,2007,21,22,1312,1536,19,21,1142,1268,20,20,1800,1978,21,25,1682,2640,21,24,1965,2793,26,26,1864,2408,17,21,783,873", :level=>:error}
But I see that event in the logstash.stdout file:
# grep '2015-10-20T04:57' -a logstash.stdout
"@timestamp" => "2015-10-20T04:57:00.000Z",
Checking the debug log for total DOY occurences:
# grep '"DOY"' -a logstash.stdout | wc -l
7974
I checked in ES and the dropped event was really dropped:
# curl -XGET 'http://localhost:9200/_search?q=@timestamp:"2015-10-20T04:57:00.000Z"'
{"took":13,"timed_out":false,"_shards":{"total":286,"successful":286,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}
So everything made it to logstash using logstash-forwarder but filebeat failed to deliver all of the events for some reason.