Kibana missing chunks of data by time

Hey all. New user here. Trying out Kibana, particularly with a focus on X-pack features. Right now, I'm running into an issue when I visualize my data. It seems that I'm missing chunks of hours from my data, which spans the course of two weeks.

In particular, my Logstash config specifies date as "yyyy-MM-dd'T'H:m". Dates can look like 2017-10-10T15:0 to indicate 3PM on October 10, or 2017-10-10T2:0 to indicate 2AM on October 10.

The data I'm looking at exists for every 10 minute segment in the original .csv file, meaning you'll see 2017-10-10T15:0, 2017-10-10T15:10, 2017-10-10T15:20.... and so forth, from October 10 to October 23. However, when I query and/or visualize my data in Kibana, I seem to be missing data around the 4PM-3AM mark every night, give or take 30 minutes. Here's what a sample of the data looks like in the Kibana Visualize tab (y-axis is count of records at each time slot):

As you can see, we're missing data from 10/10 4:30PM to 10/11 3AM, and then again from 10/11 4:30PM to 10/12 3AM. However, screenshotting my .csv, the data clearly exists at those times.

Does anybody know why the data is missing for these evenings? It definitely exists in the .csv, but for some reason, Kibana isn't detecting it. It's causing my time series to have these unexplained gaps. I'm not sure if it has to do with how timestamps are being parsed or so. I waited until the data finished indexing before proceeding with visualization and any machine learning in X-pack.

What do the events look like in Kibana? Is the data parsed correctly? Does the @timestamp match the timestamp from the csv file?

@timestamp matches the csv timestamp perfectly. May you clarify your other two questions? The data that clearly exists in the .csv just doesn't seem to appear in Kibana.

Can you show an example of an event where the @timestamp field matches the csv timestamp? What is the mapping for the @timestamp field?

Elasticsearch and Kibana requires all timestamps to be in UTC time zone. Is this the case for your events?

Here's a screenshot from the Discover tab of both @timestamp and timestamp columns.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.