Kibana4: How to set "EventTime" as time field instead of @timestamp

sunilmchaudhari · July 16, 2015, 6:05am

Hi,
I am using Kibana4.
I have one customized time field in db ("EventTime") and @timestamp as default entry time.
EventTime the timestamp when even was created in the log file at source.
I want EventTime as timefiled so that user can select time in quick view as per the log file.

Can anybody help me to bring down the delay happening in EventTime and entry.time (@timestamp) ?
I expect it to happen in real time. There is Shipper-->broker-->Indexer->nGinx in between LSF and ES.

thanks
Sunil

tbragin · July 16, 2015, 5:58pm

Hi Sunil - The question you posed is not completely clear to me. What do you mean by the "bring down the delay"? Could you elaborate?

sunilmchaudhari · July 17, 2015, 3:15am

Hello Tanya,

about delay part:
see for example below log in application log file:

2015-07-16 10:08:49 ERROR [] [org.springframework.security.web.FilterChainProxy] - Some Error with statctrace

I called this timestamp as event log time. I am storing it under the customized field "eventTime". There is another time (@timestamp) created internally which is nothing but the entry time of the event in the datastore.

Ideally this time should be almost same (may be the diff in only few seconds). But in my system, the difference is in minutes. This might be due to middlewares like shipper, broker indexer, load balancers.
Sometimes the log created on 13th July stored on 14th July. this may happen at the time of date change (around midnight). This is the delay I am talking about. And I want this delay to be as minimum as possible.

That's why I want event log time as new field and the calendar should work on this field only, so that user can search the error logs correctly by selecting most possible combination of from and to date with timestamp.
Example: to search the stacktrace of error above, he will select the date 16th july from calendar and timestamp in between 10:07:00 and 10:09:00 .

br,
Sunil

tbragin · July 17, 2015, 3:33am

Thanks for the additional detail.

You should be able to accomplish this by setting "EventTime" as the Time-Field Name - this is the field that will be used to filter using the default time-picker.

Regarding how to bring down the delay, I'd consider posting this question in the Logstash forum - I agree with you that it's likely caused by some component between LSF and ES

sunilmchaudhari · July 17, 2015, 4:19am

Hi Tanya,
Thanks for quick reply.
I can see only @timestamp field not any other field. Do I need to set any extra property while creating that field in filter?

tbragin · July 17, 2015, 4:36am

It just has to be of type "date" in your mappings -- then it should automatically show up.

sunilmchaudhari · July 17, 2015, 6:40am

Hi Tanya,
I have created one field evenTime and its being created now. But I cant see it in the left pane. Plz see screenshot which shows highlighted customized field under '_source' but not in left pane field list. why so?

br,
Sunil