ES timestamp index not working while creating index

[chakriv (chakri)]

Elastic search index template

// {
"order": 0,
"template": "jobbot",
"settings": {},
"mappings": {
"doc": {
"properties": {
"request": {
"type": "keyword"
},
"timestamp": {
"type": "date" ,
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
}

            }
        }
    },
 "aliases": {}

} //

sample entry (record in ES/kibana):

//{
"_index": "jobbot",
"_type": "doc",
"_id": "2018-05-16 14:02:02.601520774 +0000 UTC m=+61284.555636238",
"_version": 1,
"_score": 1,
"_source": {
"timestamp": "2018-05-16 14:02:02.601520774 +0000 UTC m=+61284.555636238",
"request": "individualreq"
}
}//

how do i create timestamp field for each entry and have search results by timestamp configured?

thanks
Chakri

[jscubida (J C)]

It looks like you have a mapping that isn't able to handle the timestamp in your sample entry, which includes fractions of a second and additional information after the timezone. It might be easier to format your timestamp data as one of the built-in formats. Otherwise, you'll need to revise the mapping you have. See this for more formatting info. Once you have the correct format, Kibana should recognize that timestamp is a valid time field.

[chakriv (chakri)]

thanks much.. yeah i figured date format is causing this..however when i try to re-index with adding metadata field for timestamp using logstash filters..its taking current time as timestamp not the timestamp field in message..any way to extract that out?

filter {

mutate {
add_field => { "[@metadata][source][timestamp]" => "[@metadata][timestamp]" }
}

}

after conversion data json:

{
"_index": "jobbot-2018.06.01",
"_type": "doc",
"_id": "Uvnwu2MBclkPHF46kSv5",
"_version": 1,
"_score": 2,
"_source": {
"@version": "1",
"timestamp": "2018-05-21 17:47:50.605260774 +0000 UTC m=+330187.478365347",
"@timestamp": "2018-06-01T15:21:32.046Z",
"request": "Usage"
},
"fields": {
"@timestamp": [
"2018-06-01T15:21:32.046Z"
]
}
}

before conversion json data:

{
"_index": "sejobbot",
"_type": "doc",
"_id": "2018-05-16 14:02:02.601520774 +0000 UTC m=+61284.555636238",
"_version": 1,
"_score": 1,
"_source": {
"timestamp": "2018-05-16 14:02:02.601520774 +0000 UTC m=+61284.555636238",
"request": "individualContributor"
}
}

[jscubida (J C)]

I believe Logstash is inserting the current time based on this answer. Since your timestamp field is a string upon being input into Logstash, I think you'll have to format the string and convert it into a date. This should be of assistance.