ES 6.5.2 Numeric range query returning results out of range

Elastic Stack Elasticsearch
1

Hi Team,

The field is definitely of the expected type which is float - this is the mapping of the index:

...
    "properties" : {
	....
      "utc_ts" : {
        "type" : "float"
      }
    }
  }
},

The field values are also not too large I believe - here's some example of the values of the utc_ts field:

January 4th 2019, 06:07:31.920	1,546,582,050.222
January 4th 2019, 06:07:31.920	1,546,582,051.623
January 4th 2019, 06:07:33.174	1,546,582,052.946
January 4th 2019, 06:07:33.683	1,546,582,053.42
January 4th 2019, 06:07:34.775	1,546,582,054.337

Now if I apply a range query either on Kibana or directly via ES API - the results returned do not comply to the range specified. I tried several ranges and the results are similarly incorrect.

Strange thing is I've got other similar float fields in other index patterns for which the range query works just fine.

So the problem here is I'm not sure what else I should check in order to debug this. Usually it is the wrong mapping or values bigger than the allowed type. Am I missing something?

Cheers,

2

Are all your mapping the same? How does your range query look like? Do you run it as a stand alone query or it is part of some bigger query? Can you reproduce it on a separate small index and post the recreation here?

3

Hi Igor, agreed, sorry.

Create mapping:

curl -H 'Content-Type: application/json' -XPUT 'localhost:9200/_template/test-stats?pretty' -d'
{
  "order" : 500,
  "index_patterns" : [
    "test-stats*"
  ],
  "mappings" : {
    "metrics" : {
      "properties" : {
        "double_field" : {
          "type" : "double"
        },
        "float_field" : {
          "type" : "float"
        },
        "small_float_field" : {
          "type" : "float"
        }
      }
    }
  }
}'

Then pump in some test data:

curl -X POST "localhost:9200/test-stats/metrics/_bulk?pretty" -H 'Content-Type: application/json' -d'
{ "index" : { "_id" : "1" } }
{ "float_field" : 1000000000.100, "small_float_field" : 100.100, "double_field" : 1000000000.100 }
{ "index" : { "_id" : "2" } }
{ "float_field" : 1000000011.200, "small_float_field" : 200.100, "double_field" : 1000000011.200 }
{ "index" : { "_id" : "3" } }
{ "float_field" : 1000000022.300, "small_float_field" : 300.100, "double_field" : 1000000022.300 }
{ "index" : { "_id" : "4" } }
{ "float_field" : 1000000033.400, "small_float_field" : 400.100, "double_field" : 1000000033.400 }
{ "index" : { "_id" : "5" } }
{ "float_field" : 1000000044.500, "small_float_field" : 500.100, "double_field" : 1000000044.500 }
'

My findings:

  • The problem only happens with large floats but I am not sure why because as you can see these float values are not too large or are they?
  • For small float values and double values, the range query works fine.

These are the range queries for small float values and double values which work fine:

curl -X GET "localhost:9200/test-stats/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "range" : {
            "small_float_field" : {
                "gte" : 100,
                "lte" : 400
            }
        }
    }
}
'
Output 
{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 3,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000112E9,
          "small_float_field" : 200.1,
          "double_field" : 1.0000000112E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000001E9,
          "small_float_field" : 100.1,
          "double_field" : 1.0000000001E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000223E9,
          "small_float_field" : 300.1,
          "double_field" : 1.0000000223E9
        }
      }
    ]
  }
}

curl -X GET "localhost:9200/test-stats/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "range" : {
            "double_field" : {
                "gte" : 1000000020,
                "lte" : 1000000040
            }
        }
    }
}
'
Output 
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000334E9,
          "small_float_field" : 400.1,
          "double_field" : 1.0000000334E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000223E9,
          "small_float_field" : 300.1,
          "double_field" : 1.0000000223E9
        }
      }
    ]
  }
}

But the range query for the large float field (which should return exactly the same results as the range query done for the double_field) returns results outside of the range:

curl -X GET "localhost:9200/test-stats/_search?pretty" -H 'Content-Type: application/json' -d'
{
    "query": {
        "range" : {
            "float_field" : {
                "gte" : 1000000020,
                "lte" : 1000000040
            }
        }
    }
}
'
Output 
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 5,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "5",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000445E9,
          "small_float_field" : 500.1,
          "double_field" : 1.0000000445E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000112E9,
          "small_float_field" : 200.1,
          "double_field" : 1.0000000112E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000334E9,
          "small_float_field" : 400.1,
          "double_field" : 1.0000000334E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000001E9,
          "small_float_field" : 100.1,
          "double_field" : 1.0000000001E9
        }
      },
      {
        "_index" : "test-stats",
        "_type" : "metrics",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "float_field" : 1.0000000223E9,
          "small_float_field" : 300.1,
          "double_field" : 1.0000000223E9
        }
      }
    ]
  }
}
4

I've looked into this further and it looks like my float values are indeed too big - I did not realise that float numbers allow only up to 9 significant decimal digits and my values have more than that.

It threw me off because on ES and Kibana I can still see the values represented correctly, i.e. the values do not look rounded on Kibana. Both double and float values look the same so I naturally expected the comparison / range query to work the same for both. I got bamboozled again.

image
image.png1127×448 15.9 KB

5

Yes, it is basically not enough precision in float to represent that. If you take a look at the actual values, you have pretty much only 2 values there 1000000000 and 1000000064

GET test-stats/_search
{
  "size": 0,
  "aggs": {
    "float_values": {
      "terms": {
        "field": "float_field",
        "size": 10
      },
      "aggs": {
        "values": {
          "top_hits": {
            "size": 10,
            "_source": {
              "includes": "float_field"
            }
          }
        }
      }
    }
  }
}

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 5,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "float_values": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": 1000000000,
          "doc_count": 3,
          "values": {
            "hits": {
              "total": 3,
              "max_score": 1,
              "hits": [
                {
                  "_index": "test-stats",
                  "_type": "metrics",
                  "_id": "2",
                  "_score": 1,
                  "_source": {
                    "float_field": 1000000011.2
                  }
                },
                {
                  "_index": "test-stats",
                  "_type": "metrics",
                  "_id": "1",
                  "_score": 1,
                  "_source": {
                    "float_field": 1000000000.1
                  }
                },
                {
                  "_index": "test-stats",
                  "_type": "metrics",
                  "_id": "3",
                  "_score": 1,
                  "_source": {
                    "float_field": 1000000022.3
                  }
                }
              ]
            }
          }
        },
        {
          "key": 1000000064,
          "doc_count": 2,
          "values": {
            "hits": {
              "total": 2,
              "max_score": 1,
              "hits": [
                {
                  "_index": "test-stats",
                  "_type": "metrics",
                  "_id": "5",
                  "_score": 1,
                  "_source": {
                    "float_field": 1000000044.5
                  }
                },
                {
                  "_index": "test-stats",
                  "_type": "metrics",
                  "_id": "4",
                  "_score": 1,
                  "_source": {
                    "float_field": 1000000033.4
                  }
                }
              ]
            }
          }
        }
      ]
    }
  }
}

And your query translates into

{
  "_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
  },
  "valid": true,
  "explanations": [
    {
      "index": "test-stats",
      "valid": true,
      "explanation": "float_field:[1.0E9 TO 1.00000006E9]"
    }
  ]
}

The values that you see in Kibana are retrieved from source as double values. So you get higher precision there.

1 Like
6

Thanks Igor

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.