No data when custom timestamp is used for Time-field while adding new index

[Ruthu_Ramya]

Hi,

In logstash, I am parsing timestamps from the log lines:

Jan 30,12:49:56.785 INFO message 1
Jan 30,12:49:56.788 INFO message 2

and storing in the field

add_field => {"log_timestamp" => "%{[@metadata][month]} %{[@metadata][day]},%{[@metadata][time]}"}

Using datefilter, it is stored as a date:

date {
match => [ "log_timestamp", "MMM dd,HH:mm:ss.SSS"]
target => "log_timestamp"
}

So, in elasticsearch,
I have log_timestamp as:

2017-01-30T12:49:56.785Z
2017-01-30T12:49:56.788Z

& @timestamp (log file parse time:auto calculated) as:

2017-03-08T10:14:58.639Z
2017-03-08T10:14:58.639Z

If I select log_timestamp while creating a new index, no data is shown in Kibana. If I use @timestamp, the data is visible.

Now, if I use this date filter to store the timestamp in the log to @timestamp,
I have @timestamp as:

2017-01-30T12:49:56.785Z
2017-01-30T12:49:56.788Z

This time, using @timestamp during index creation will show no data

I am not sure what I am missing here or doing incorrect. Should I enable some flag or something to indicate Kibana to accept my calculated time-field? Suggestions please. Thanks.

Regards,
Ruthu

[weltenwort (Felix Stürmer)]

Hi @Ruthu_Ramya,

this might be a timezone issue. Without a timezone offset Elasticsearch assumes the dates to be in UTC. In Kibana, the time picker automatically uses the local timezone as deduced from the browser environment. So if the dates you parse from the log file are not in UTC, it might be useful to add a timezone offset to the date string.

[Ruthu_Ramya]

Hi @weltenwort,

I added a timezone to the Date filter.

timezone => "Asia/Jerusalem"

The problem still exists. Is this what you were referring to? If it is something different, could you please quote an example. Thanks.

[Ruthu_Ramya]

Found the cause. In kibana, "Past 15 minutes" is the default discover. Since my timestamp from log was from January, it never got selected. Hence no data was loaded for the index.