I have two logstash servers that writes to Elasticsearch. Filebeat reads logs from file and sends them to these logstash servers. I am conducting a test for failover/outage and also what will happen if both the logstash servers are down. This is how the test was conducted.
- Stopped both Logstash servers. October 7 11:00
- Started Filebeat on multiple clients. October 7 11:15
- Started the application that generates the logs. October 7 11:20
- Kept the logstash servers off for 12 hours to mimic an outage of 12 hours and then bring them back on. October 8 00:20
- Stopped all Filebeat instances October 8 11:00
Now when I check Kibana to see what time the logs were sent. I see there are spikes right after October 8 00:20 which makes sense as all the logs will be coming in to Elasticsearch when the logstash servers have started but what's odd is that Kibana shows that some logs were inserted between October 7 12:00 and October 7 15:00. Logstash servers were not even ON at this time. I am using the @timestamp field. Am I missing something? How are the logs sent at time when Logstash servers are not even running?