I just started using Logstash along with Elasticsearch and Kibana to visualize some log data coming from a database, and I am running into an issue with what I believe to be the date filter. Currently I am parsing the log line with the dissect filter, which parses the date section out as expected. The date filter then changes the @timestamp field to the date printed in the log message.
I am testing now on a 4000 line excerpt from the real file. There is a log message printed every 1 hour, 24 hours a day, 7 days a week, except for one hour that wasn't logged for whatever reason. Just one line removed from the file, essentially. When I feed this data into Logstash, everything up to the missing line is ingested exactly as expected, and Kibana displays it perfectly. However, all of the entries after the missing line have their timestamp offset an hour too late. So Kibana shows the time as a certain hour, but all the data for that hour is actually from the previous hour. I can't figure out why this would be the case.
To make it more clear: March 12, 2017 at 01:00:00 was logged. The next log entry is for March 12, 2017 at 03:00:00. However, Kibana displays this as having occured at 04:00:00. It offsets the timestamp like this for all successive values. If I zoom into a visualization, the x-axis goes 01:00:00, 03:00:00, 04:00:00 - its completely missing 02:00:00 (the missing line)! But doesn't have a point at 03:00:00 either! It skips straight to 4! Really scratching my head over this one, I can't figure out why this would be the case. Thanks for any help!
I am on Elasticsearch, Logstash, and Kibana 5.4.1