Filtering "long" type values returns 0 hits, no results


(Jagden) #1

Hi,

I'm having an index with mapping as following:

domainId":{"type":"long"}

In kibana, I see values for this field, for example:

7,653,256,037,708,803,072
7,055,736,606,716,153,856
-3,956,219,722,791,012,864
.
.

The values are coming from a long field in Java and are inserted to ES via the Java API.

When doing a filter search in Kibana to the domainId field and searching for a specific value, I always get 0 results.
Also, when doing the query via the GET URL, I also get no results.
One thing I've noticed is that for small values, I do get results.
Any idea what may cause this behavior ?

http://x.x.x.x:9200/index_2016.06.16/_search?pretty=true&q=domainId:-3956219722791012864

Also other "match" queries for the domainId field are not working either.

The only solution I'm thinking of, is to index the field as a string, but only as a last resort.

Please advice.


(Alexander Reelsen) #2

Hey,

have a tried a query using the body of the HTTP reuqest, like this

GET company/_search
{
  "query": {
    "term": {
      "domainId": "-3956219722791012864"
    }
  }
}


(Jagden) #3

Yes, no results either :frowning:


(Alexander Reelsen) #4

hey,

can you provide an example, that everyone can reproduce, so debugging gets easier?

--Alex


(Jagden) #5

It's a little hard to since it's production data.

But - found the following problem, any idea what can cause this ?

Doing query to bring all entries in Kibana, shows an entry as following:

t_id: AVVZfuohIBNZBuj1MuJa
domainId: 8,797,013,729,664,655,360

Going to elasticsearch and doing query against the _id values gives the following from elastic:

{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_id" : "AVVZfuohIBNZBuj1MuJa",
"_score" : 1.0,
"_source" : {
"domainId" : 8797013729664655321,
}
} ]
}
}

See the difference ?? Elastic shows: 8797013729664655321 while Kibana shows: 8797013729664655360

Any idea what may cause this ??


(Jagden) #6

Maybe it's this ?


(Alexander Reelsen) #7

have you tried changing the formatter as mentioned in the issue, to verify that this is the problem?


(Jagden) #8

I have tried to play with the formatting of the field according to these posts but none of them seems to work.
At the moment I see no solution to this problem other than changing the field type to string.


(Flavio Pompermaier) #9

Have you found a solution to this? Could it be related to ignore_above setting? I'm also experiencing the same problem...


(Jonas) #10

Same issue here. We don't use Kibana as frontend but only access Elasticsearch programmatically.

I have a document that looks like this when it's returned from a query:

{
    "took": 3,
    "timed_out": false,
    "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
    },
    "hits": {
        "total": 1,
        "max_score": 1,
        "hits": [
            {
                "_index": "datindex-2016",
                "_type": "doc",
                "_id": "85429229",
                "_score": 1,
                "_source": {                    
                    "FID": 7352157926092659000                    
                }
            }
        ]
    }
}

When I query the index by this specific 'FID' I get no result. Query looks like:

{
  "query": {
    "query_string": {       
      "query": "FID:7352157926092659000"
    }
   }
}

I checked the source system and found out that the value that has been sent to Elasticsearch has been 7352157926092658501 (last four digits are different). When I use the original value to query the index I get one result:

{
  "query": {
    "query_string": {       
      "query": "FID:7352157926092658501"
    }
   }
}

I'm using version 5.1.2. Any help is appreciated. Thanks!


(Jonas) #11

Looks like nobody is looking into this but nevertheless I want to share what I've found out:

Seems like it is related to the fact that every number in JavaScript is internally stored as "Double" with 53-Bit mantissa. So every large number that doesn't fit into these 53 bits is truncated which explaines the deviation. Worst case is that you have more than one large value which is truncated to the same 53-bit value.

Looks like the only solution is to index such values as strings. I would have expected some kind of error message when inserting such values as numbers into an index...

If you want to try it out by yourself just open a JavaScript console in your browser:

var a = 7352157926092658505;
var b = 7352157926092658504;
a === b;

(Christian Dahlqvist) #12

You may be able to send them as strings over JSON but still map them as numbers in Elasticsearch.


(Jonas) #13

Let's assume I do this: Which value do I see when I query the document via REST api? As the value is stored as number I would assume that I see the (wrong) truncated value again. If the Elasticsearch stack is using JavaScript for the mapping it is stored incorrectly either.

I tried it out:

I've created an index:

{   
  "mappings": {
	  "doc": {
  		  "properties": {
    		  "largeNumber": { "type": "long" }
  		  }
	  }
  }
}

Then I've inserted a document:

{ 
  "largeNumber": "7352157926092658505",
  "largeNumberWithoutString": 7352157926092658505,
  "asString": "7352157926092658505" 
}

Query result is interesting:

"_source": {
  "largeNumber": "7352157926092658505",
  "largeNumberWithoutString": 7352157926092659000,
  "asString": "7352157926092658505"
}

I wouldn't have expected the quotation marks for field 'largeNumber' but it seems to work like that.