Discover: Field data loading is forbidden on [loggingTime]

[Honda_fred_elk (Fredrick Yessaian)]

Hello,

I use Filebeat to export JSON String to --> logstash --> elasticsearch --> Kibana

My application's log4j creates each log as JSON String like below.. it has loggingTime key & pair which is actual timestamp of that log message.

{"loggingTime":"Wed 14 Jun 2017 15-03-49 126 EDT","applicationName":"LOT_CONTROL_SIM_2","hostName":"VNWQ63213.hmin.am.honda.com","clientName":"LOT_CONTROL_SIM_2","loglevel":"INFO","logMessage":"Successfully played clip: Buzzer.wav"}

Logstash adds @timestamp in which there is a difference between actual loggingTime and @timestamp.. so I don't want to use it. logically @timestamp is a message received timestamp from logstash.

{"type":"client_log","loggingTime":"Wed 14 Jun 2017 15-03-49 126 EDT","applicationName":"LOT_CONTROL_SIM_2","@timestamp":"2017-06-14T19:03:54.338Z","beat":{"name":"VNWQ63213","hostname":"VNWQ63213","version":"5.1.2"},"loglevel":"INFO","source":"/read-write/mnt/addons/GALCClientLogs/client-jsonoutput.log","offset":4157264,"logMessage":"Successfully played clip: Buzzer.wav","hostName":"VNWQ63213.hmin.am.honda.com","clientName":"LOT_CONTROL_SIM_2","input_type":"log","@version":"1","host":"VNWQ63213","tags":["beats","beats_input_raw_event"]}

But when I see the output through Kibana.. I see loggingTime as string type. Also When I tried to sort loggingTime field in Discover page in Kibana I'm getting below error and I am not able to sort the logs based on loggingTime field

Discover: Field data loading is forbidden on [loggingTime]

I want to have loggingTime field as as sortable field based on its timestamp. This will add value to the users, when they try to connect actual logs with timestamp to narrow down for issues.

So I want to have loggingTime field as sortable in kibana, How can I achieve this?

Thanks
Fredrick

[Honda_fred_elk (Fredrick Yessaian)]

I read that It could be done by overriding @timestamp through logstash filter plugin date.. I solved this issue by having following settings in the logstash config file

filter{
date{
locale => "en"
match => ["loggingTime","EEE d MMM yyyy HH-mm-ss SSS z"]
target => "@timestamp"
}
}

After this .. I can sort time column is nothing but @timestamp field in Kibana .. and verified both values are same..
My problem is resolved