Kibana 4.1.1 unable to map timestamp which is in UNIX Epoch format


(Naren) #1

Hi Team,

I have recently upgraded my Kibana from 3 to 4.1.1, my nginx access logs has Epoch Time format like 1434589605.318 and kibana is not able to detect this time stamp field, hence I am not able to segregate the logs, however if I try the same access logs on kibana3 it is able to detect timestamp after I mention the filed in Time picker.

Please guide me on how to map this time stamp to index so that I can segregate the logs based on time stamp.

Thanks and Regards,
Naren.


(Mark Walkom) #2

What does the mapping in ES look like for that field?


(Tanya Bragin) #3

There is no change in the way Kibana 4 treats timestamps, compared to Kibana 3. Both require that your time field is mapped as a "date" in Elasticsearch.

For reference, here is a sample mapping that works:

  "@timestamp" : {
    "type" : "date",
    "format" : "strict_date_optional_time||epoch_millis"
  },

(Naren) #4

Thanks for your reply, my kibana4 gets logs from one ES cluster where time format is in epoch format and it is mapped to field "ts", however nothing is being displayed in indices filed

could you please guide me on where should I map the ts field so that it will populate in Indices

Thanks and Regards,
Narendra.


(Tanya Bragin) #5

Could you post your mappings from ES to make sure we're talking about the same thing?


(Naren) #6

Hi Tanya,

Thanks for your reply, below are my mappings.


(Tanya Bragin) #7

So, I don't see anything with type "date" in there -- that's what you need.


(Naren) #8

The Field TS is the timestamp in epoch format, in kiabna 3 I am able to mention @ts in time picker and it is able to recognise it, however in kibana4 i am not able to map it,

Is there a way to mention it in kibana4 apart from modifying it in elastic search?


#9

Hello!
I've been struggling with timestamps too. The only way i found to get it to work as i wanted to, was to create a new template for timestamp mapping.
curl -XPUT 'http://YOURELASTICSEARCHSERVER:9200/_template/indexname*/' -d @elastic_template.json

content of file "elastic_template.json"
{
"template": "indexname*",
"order": 1,
"mappings" : {
"index" : {
"properties" : {
"log_date" : {
"type" : "date",
"format": "yyyy-MM-dd HH:mm:ss,SSS||yyyy-MM-dd HH:mm:ss.SSS||yyyy/MM/dd HH:mm:ss||yyyy-MM-dd||MMM dd, yyyy HH:mm:ss a"
},
"appname" : {
"type" : "string",
"index": "not_analyzed"
},
"msg_content" : {
"type" : "string"
}
}
}
}
}
}

I guess this will work for epoch_millis as well. Most important thing: Ensure elasticsearch is saving your timestamp as date, check with:
curl -XGET 'http://YOURELASTICSEARCHSERVER:9200/indexname*/_mapping?pretty'

Hope it helps!


(system) #10