# Long JSON number lost precision

**URL:** <https://discuss.elastic.co/t/long-json-number-lost-precision/72124>\
**Category:** Elasticsearch\
**Created:** [January 19, 2017, 7:11am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124 "2017-01-19T07:11:33Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 19, 2017, 7:11am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/1 "2017-01-19T07:11:33Z")

</div>

Elasticsearch Version:  
v5.0.2

Index Info:

> GET /\_cat/indices/dblog-2017.01.19?v

> health status index uuid pri rep docs.count docs.deleted store.size pri.store.size  
> green open dblog-2017.01.19 Lis\_GE3nTVutbUZMJYXDZw 10 1 1976053299 4 5.3tb 2.6tb

Problem Description:  
When searching on a keyword field _traceId_ with TermQuery, for example:  
\> POST /dblog-2017.01.19/\_search  
\> {  
\> "query": {  
\> "term": {  
\> "traceId": {  
\> "value": "6226230315557965000"  
\> }  
\> }  
\> },  
\> "\_source": "traceId"  
\> }

It returned no hits.

If using prefix query (& wildcard query) , the doc can be matched when the value queried is less than or equal to 15 characters like below:

> POST /dblog-2017.01.19/\_search  
> {  
> "query": {  
> "prefix": {  
> "traceId": {  
> "value": "622623031555796"  
> }  
> }  
> },  
> "\_source": "traceId"  
> }

```
{
      "took": 4,
      "timed_out": false,
      "_shards": {
        "total": 10,
        "successful": 10,
        "failed": 0
      },
      "hits": {
        "total": 1,
        "max_score": 1,
        "hits": [
          {
            "_index": "dblog-2017.01.19",
            "_type": "logs",
            "_id": "AVm0Cv1DARfp1s5Erl20",
            "_score": 1,
            "_source": {
              "traceId": 6226230315557965000
            }
          }
        ]
      }
    }

```

If I index only a few docs to a test index, the same TermQuery matches the doc correctly.

I am not sure if this is a bug or constrain of Elasticsearch/Lucene. As the traceID field is of very high cardinality, and the index is with almost 2 Billions of docs, they looked like contributed to the problem.

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [January 19, 2017, 4:47pm UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/2 "2017-01-19T16:47:37Z")

</div>

ES/Lucene should support very high cardinality fields just fine, so this sounds like a possible bug.

Are many of your id values affected, or just a small subset?

Can you figure out which shard this document was routed to and zip up that entire Lucene index and post somewhere? I can pull it down and try to dig into it.

Mike McCandless

---

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 20, 2017, 4:58am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/3 "2017-01-20T04:58:00Z")

</div>

Hi, Mike

Thanks a lot for responding!

When it happened, all id values are affected. I built a test index with one shard and 21,000 sample docs, on which the problem can be reproduced. The index files can be downloaded from  
[https://drive.google.com/file/d/0B6UVCJccYns8VVpwYURTbTlSMGM/view?usp=sharing](https://drive.google.com/file/d/0B6UVCJccYns8VVpwYURTbTlSMGM/view?usp=sharing)

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [January 20, 2017, 11:57am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/4 "2017-01-20T11:57:25Z")

</div>

Thanks, I was able to download the index ... I'll dig into it.

Mike McCandless

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [January 20, 2017, 1:42pm UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/5 "2017-01-20T13:42:17Z")

</div>

The index has 21,000 documents, no deletions.

It has 20,154 unique `traceId` (i.e. some documents share `traceId`).

Then I made a small java app to count number of docs for each `traceId`:

```auto
import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.nio.file.Path;
import java.nio.file.Paths;

import org.apache.lucene.index.DirectoryReader;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.FSDirectory;

// javac -cp ../build/core/lucene-core-6.2.0-SNAPSHOT.jar TraceID.java; java -cp .:../build/core/lucene-core-6.2.0-SNAPSHOT.jar TraceID

public class TraceID {
  public static void main(String[] args) throws IOException {
    Path path = Paths.get("broken");
    Directory dir = FSDirectory.open(path);
    IndexReader r = DirectoryReader.open(dir);
    System.out.println("maxDoc=" + r.maxDoc() + " numDocs=" + r.numDocs());
    IndexSearcher s = new IndexSearcher(r);
    try (BufferedReader br = new BufferedReader(new FileReader(new File("/l/62/traceIds.txt")))) {
      String line;
      while((line = br.readLine()) != null) {
        int count = s.count(new TermQuery(new Term("traceId", line.trim())));
        if (count != 1) {
          System.out.println("GOT: " + count + " for traceId=" + line.trim());
        }
      }
    }
    r.close();
    dir.close();
  }
}

```

And while some of the `traceId`s had \> 1 count, none of them had count 0. So as far as I can tell, the Lucene side is working correctly.

Also, most `traceId`s are length 19, but some are length 15-18 as well. Is that expected?

Do you have an example `traceId` that fails to come back for this index?

Mike McCandless

---

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 22, 2017, 6:30am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/6 "2017-01-22T06:30:54Z")

</div>

High cardinality and docs count are red herring. After probing a bit more today, it looked the json parser used by elasticsearch rounded a json long value when converting it to a string.

Below screen shot illustrates what is the problem. The traceId in \_source is already rounded. This's why the attempt to filter by traceId from source always failed. However the terms aggregation still shows the correct key.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/d/deb7ee003e5a211419c5e2394df41f01a62031e9.png)

I also tested defining traceId as long in elasticsearch. In this scenario , both source and aggregation key lost precision.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/e/e26e2fa9f3602c63753e4a109dba999ff401e01d.png)

Kenny Wu

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [January 24, 2017, 12:56am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/7 "2017-01-24T00:56:09Z")

</div>

Hmm I just tested ES 5.1.2 using your first example but the value is correct (not truncated).

Can you test ES 5.1.2? Maybe there was a bug that's been fixed.

Mike McCandless

---

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 24, 2017, 1:36am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/8 "2017-01-24T01:36:09Z")

</div>

I tested ES 5.1.2 and the problem remains. Please be aware traceId from source json is not a string but of type number, ie, { "traceId": `1026314602185330712` } . It looks when parsing this long number, the precision is lost.

```
GET /

{
  "name": "cAewfKa",
  "cluster_name": "kenny_test",
  "cluster_uuid": "JUY_mFLJSOmwCfxCZRPdsA",
  "version": {
    "number": "5.1.2",
    "build_hash": "c8c4c16",
    "build_date": "2017-01-11T20:18:39.146Z",
    "build_snapshot": false,
    "lucene_version": "6.3.0"
  },
  "tagline": "You Know, for Search"
}

```

* * *

```
GET /test_me/_mappings

{
  "test_me": {
    "mappings": {
      "logs": {
        "properties": {
          "traceId": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

```

* * *

```
POST /test_me/logs
{
  "traceId": 1026314602185330712
}

```

* * *

```
POST /test_me/_search

 {
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "failed": 0
  },
  "hits": {
    "total": 1,
    "max_score": 1,
    "hits": [
      {
        "_index": "test_me",
        "_type": "logs",
        "_id": "AVnOGVpfDdPNLqd8e68H",
        "_score": 1,
        "_source": {
          "traceId": 1026314602185330700
        }
      }
    ]
  }
}
```

---

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 24, 2017, 1:48am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/9 "2017-01-24T01:48:38Z")

</div>

I changed the title of this post as the original one is misleading.

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [January 24, 2017, 2:40am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/10 "2017-01-24T02:40:02Z")

</div>

Hmm, how are you submitting your JSON requests? It works for me with ES 5.1.2 when I use `curl`:

```auto
curl -XPUT 'localhost:9200/test_me?pretty' -d'
{
  "mappings": {
    "logs": {
      "properties": {
        "traceId": {
	  "type": "keyword"
	}
      }
    }
  }
}
'
-->
{
  "acknowledged" : true,
  "shards_acknowledged" : true
}

curl -XPOST 'localhost:9200/test_me/logs?pretty' -d'
{
  "traceId": 1026314602185332762
}
'
-->
{
  "_index" : "test_me",
  "_type" : "logs",
  "_id" : "AVnOUV3bXuDudHHhdSuV",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "created" : true
}

curl -XPOST 'localhost:9200/test_me/logs/_search?pretty' -d'
{
  "aggs": {
    "test": {
      "terms": {
        "field": "traceId",
        "size": 10
      }
    }
  }
}
'
->
{
  "took" : 27,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "test_me",
        "_type" : "logs",
        "_id" : "AVnOUV3bXuDudHHhdSuV",
        "_score" : 1.0,
        "_source" : {
          "traceId" : 1026314602185332762
        }
      }
    ]
  },
  "aggregations" : {
    "test" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "1026314602185332762",
          "doc_count" : 1
        }
      ]
    }
  }
}

```

Mike McCandless

---

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 24, 2017, 3:26am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/11 "2017-01-24T03:26:27Z")

</div>

I just used curl for the same request and can confirm the result is good.

Previous tests I did are under Dev console (sense) within Kibana. So this now looks to be a problem with Kibana (or the js library it used for converting json long number).

![](https://us1.discourse-cdn.com/elastic/original/2X/5/525435c78b4c8dc3450ca9a457028d56322bef13.png)

 ![](https://us1.discourse-cdn.com/elastic/original/2X/8/84d1c26f6dd26d133751d4fd01cc578ace7b8c3d.png)

---

<div class="post-metadata">

**Author:** ![mikemccand](https://avatars.discourse-cdn.com/v4/letter/m/f04885/32.png) [@mikemccand](https://discuss.elastic.co/u/mikemccand)\
**Post date:** [January 24, 2017, 12:03pm UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/12 "2017-01-24T12:03:42Z")

</div>

Ahh, I see ... so something in Kibana's dev console is maybe truncating the long value. I assume a workaround here is for you to make this long value a string instead?

Can you open a Kibana issue, linking to this discussion? Thanks.

Mike McCandless

---

<div class="post-metadata">

**Author:** ![kwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwu/32/14714_2.png) [@kwu](https://discuss.elastic.co/u/kwu)\
**Post date:** [January 25, 2017, 2:31am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/13 "2017-01-25T02:31:11Z")

</div>

Yes, I've already make this traceId a string to get around the problem. But I think Kibana needs a way to get it fixed as this not only affect Dev console but all other functionality when a long value is used for filtering and graphing. I'll raise this issue to Kibana team via Github.

Thanks very much for helping me finding the root cause!

Regards  
Kenny

---

<div class="post-metadata">

**Author:** ![mabn0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mabn0/32/15135_2.png) [@mabn0](https://discuss.elastic.co/u/mabn0)\
**Post date:** [February 3, 2017, 2:40am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/14 "2017-02-03T02:40:42Z")

</div>

I also spent some time trying to figure it out. But it's just the nature of JavaScript:

> <https://stackoverflow.com/questions/307179/what-is-javascripts-highest-integer-value-that-a-number-can-go-to-without-losin>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2017, 2:41am UTC](https://discuss.elastic.co/t/long-json-number-lost-precision/72124/15 "2017-03-03T02:41:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.