Re: Date Formatting / Sorting by Float

Elastic Stack Elasticsearch
1

For anyone interested..

Fixed the date saving by creating the mapping like;

{
"tstamp" : { "type" : "date", "format" : "basic_date_time" }
}

Then making sure I only send the unix timestamp without microseconds
(sending microseconds in a seperate field).

This fixes the sorting, but I'm still battling with "No filter registered
for [..]"...

On Wednesday, 9 July 2014 10:07:52 UTC+1, Mullac wrote:

Hi,

I have a date and a unix timestamp field in my mapping. Experiencing
several issues..

  1. I cannot get Elasticsearch to accept any kind of date format I have
    tried for the date field
  2. Sorting on the unix timestamp field is not working (which is type
    float)
  3. Unable to filter on the unix timestamp field ("No filtered
    registered for [tstamp_unix]

Example of the data going in (note the fractions of a second):

{
"tstamp" : "2014-07-09 08:33:04.178900",
"tstamp_unix" : 1404894784.178900
}

I'm aware I don't need the unix value however as the mapping just wasn't
working and date is now set to type string. In terms of mapping I have
tried:

{
"type1": {
"tstamp": {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss.S" // also yyyy-MM-dd
HH:mm:ss.SSS, y-M-d H:m:s.S etc etc
}
}
}

Then with the sorting - tstamp_unix is mapped as type float. When
performing a sort against this field, it does something very weird.. one
example, see the resulted sort value..:

// Query - all records newer than 5 minutes ago
{
"size" : 1000,
"fields" : [ "tstamp_unix", "tstamp" ],
"sort": [
{ "tstamp_unix" : {"order" : "desc"} }
],
"query" : {
"filtered": {
"query": {
"range": {
"tstamp_unix":{
"gt": 1404895819
}
}
},
"filter": {
"term": { "grouped_identifier": "some_value" }
}
}
}
}

// Result
{
"_index" : "index",
"_type" : "type",
"_id" : "XYqXjRHPR4m8ndYKyp5-8Q",
"_score" : null,
"fields" : {
"tstamp_unix" : [ "1404896082.123818" ],
"tstamp" : [ "2014-07-09 08:54:42.123818" ]
},
"sort" : [ 1.40489613E9 ]
}

Can't find any other resources on this.. any ideas??

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fddb6b46-006c-460e-99d4-3467aa653434%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Fixed filter, needed to be a boolean with two filters under must (as
array). Final query:

{
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{"term": { "grouped_identifier": "some_value" }},
{
"range": {
"tstamp_unix":{
"lt": 14044896028
},
"_cache": false
}
}
]
}
}
}
}
}

On Thursday, 10 July 2014 08:56:07 UTC+1, Mullac wrote:

For anyone interested..

Fixed the date saving by creating the mapping like;

{
"tstamp" : { "type" : "date", "format" : "basic_date_time" }
}

Then making sure I only send the unix timestamp without microseconds
(sending microseconds in a seperate field).

This fixes the sorting, but I'm still battling with "No filter registered
for [..]"...

On Wednesday, 9 July 2014 10:07:52 UTC+1, Mullac wrote:

Hi,

I have a date and a unix timestamp field in my mapping. Experiencing
several issues..

  1. I cannot get Elasticsearch to accept any kind of date format I
    have tried for the date field
  2. Sorting on the unix timestamp field is not working (which is type
    float)
  3. Unable to filter on the unix timestamp field ("No filtered
    registered for [tstamp_unix]

Example of the data going in (note the fractions of a second):

{
"tstamp" : "2014-07-09 08:33:04.178900",
"tstamp_unix" : 1404894784.178900
}

I'm aware I don't need the unix value however as the mapping just wasn't
working and date is now set to type string. In terms of mapping I have
tried:

{
"type1": {
"tstamp": {
"type" : "date",
"format" : "yyyy-MM-dd HH:mm:ss.S" // also yyyy-MM-dd
HH:mm:ss.SSS, y-M-d H:m:s.S etc etc
}
}
}

Then with the sorting - tstamp_unix is mapped as type float. When
performing a sort against this field, it does something very weird.. one
example, see the resulted sort value..:

// Query - all records newer than 5 minutes ago
{
"size" : 1000,
"fields" : [ "tstamp_unix", "tstamp" ],
"sort": [
{ "tstamp_unix" : {"order" : "desc"} }
],
"query" : {
"filtered": {
"query": {
"range": {
"tstamp_unix":{
"gt": 1404895819
}
}
},
"filter": {
"term": { "grouped_identifier": "some_value" }
}
}
}
}

// Result
{
"_index" : "index",
"_type" : "type",
"_id" : "XYqXjRHPR4m8ndYKyp5-8Q",
"_score" : null,
"fields" : {
"tstamp_unix" : [ "1404896082.123818" ],
"tstamp" : [ "2014-07-09 08:54:42.123818" ]
},
"sort" : [ 1.40489613E9 ]
}

Can't find any other resources on this.. any ideas??

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8393a657-5e14-4e7b-85fd-f61c7cf560ff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.