Today's surprise on the cluster is my logstash-forwarder nodes seeming to have miss a lot of data during night... they appeared to run normally... but the amount of logs received is a portion owhat should be there.
By restarting logstash-forwarder, I see lot of 100 entry packets being sended all the sudden to logstas server... so I updated the cron to periodically restart logstash-forwarder sice it seems to stall on logrotation, full moon or don't know what at night hehehe.
Today... I see a load of log entries at night, at a moment where activity is iddle! so it seems tons of data has arrived all of the sudden, and timestamped at system arrival time.
Is this normal?