# Scientific notation in incoming data

**URL:** <https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517>\
**Category:** Elasticsearch\
**Created:** [September 1, 2016, 8:48am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517 "2016-09-01T08:48:31Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 1, 2016, 8:48am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/1 "2016-09-01T08:48:31Z")

</div>

Dear colleagues.  
do you know the maximum possible number of fields for index ?  
In my case i have over 200 from the fluentd configured but in Kibana indexes i have Fields ( **45** ) only visible . Is there some limitation for the number of Fields ?  
Also dynamic mapping used.

```
 "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "omit_norms" : true},
      "dynamic_templates" : [ {
        "ID" : {
          "match" : "ID",
          "mapping" : {
            "type" : "string", "index" : "not_analyzed"
          }
        }
      },
      {
        "Name" : {
          "match" : "Name",
          "mapping" : {
            "type" : "string", "index" : "not_analyzed"
          }
        }
      },
      {
        "ip" : {
          "match" : "ip",
          "mapping" : {
            "type" : "ip", "index" : "not_analyzed"
          }
        }
      },
      {
        "string_fields" : {
          "match" : "*",
          "mapping" : {
            "type" : "float", "index" : "not_analyzed"
          }
        }
      }],
      "properties" : {
        "@timestamp" : { "type" : "date" }
          }
        }
      }
      
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [September 1, 2016, 10:50am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/2 "2016-09-01T10:50:58Z")

</div>

Hi Dmitriy,

In Kibana, on the Index Patterns page (under Settings), have you tried refreshing your index pattern? That should pick up any fields that are present in your Elasticsearch indices (for the chosen pattern) that are not yet known to Kibana.

Cheers,

Shaunak

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 1, 2016, 12:19pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/3 "2016-09-01T12:19:19Z")

</div>

Yes i did but same 45.  
it seems ELastic limitation ? or kibana ...

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [September 1, 2016, 9:35pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/4 "2016-09-01T21:35:39Z")

</div>

Both Elasticsearch and Kibana should be able to handle 200 fields, no problem. Can you do a `GET <index name>` Elasticsearch API call, where `<index name>` is a specific index that should contain the 200 fields, and paste the output here?

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 1, 2016, 9:56pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/5 "2016-09-01T21:56:24Z")

</div>

Ok but what is the max number of fields ?

below is the case with fluentd config like this  
keys Time,ID,Name,ip,val\_1,val\_2,val\_3,val\_4,.....,val\_112

> ```
> "dynamic_templates": [
> {
> "ID": {
> "mapping": {
> "index": "not_analyzed",
> "type": "string"
> },
> "match": "ID"
> }
> },
> {
> "Name": {
> "mapping": {
> "index": "not_analyzed",
> "type": "string"
> },
> "match": "Name"
> }
> },
> {
> "ip": {
> "mapping": {
> "index": "not_analyzed",
> "type": "ip"
> },
> "match": "ip"
> }
> },
> {
> "string_fields": {
> "mapping": {
> "index": "not_analyzed",
> "type": "float"
> },
> "match": "*"
> }
> }
> ],
> "properties": {
> "@timestamp": {
> "type": "date",
> "format": "strict_date_optional_time||epoch_millis"
> },
> "ID": {
> "type": "string",
> "index": "not_analyzed"
> },
> "Name": {
> "type": "string",
> "index": "not_analyzed"
> },
> "ip": {
> "type": "ip"
> },
> "val_1": {
> "type": "float"
> },
> "val_10": {
> "type": "float"
> },
> "val_100": {
> "type": "float"
> },
> "val_101": {
> "type": "float"
> },
> "val_102": {
> "type": "float"
> },
> "val_103": {
> "type": "float"
> },
> "val_104": {
> "type": "float"
> },
> "val_105": {
> "type": "float"
> },
> "val_106": {
> "type": "float"
> },
> "val_107": {
> "type": "float"
> },
> "val_108": {
> "type": "float"
> },
> "val_109": {
> "type": "float"
> },
> "val_11": {
> "type": "float"
> },
> "val_110": {
> "type": "float"
> },
> "val_111": {
> "type": "float"
> },
> "val_112": {
> "type": "float"
> 
> ```

in case fluentd config:

> keys Time,ID,Name,ip,MemUsageAvg(%),MemUsageMin(%),MemUsageMax(%),MemUsageTot(%),MemUsageCnt(count),CpuUsageAvg(%),CpuUsageMin(%),CpuUsageMax(%).......  
> elastic is empty -why ?

---

<div class="post-metadata">

**Author:** ![1222kkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/1222kkk/32/23635_2.png) [@1222kkk](https://discuss.elastic.co/u/1222kkk)\
**Post date:** [September 2, 2016, 1:51am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/6 "2016-09-02T01:51:28Z")

</div>

I'm concerned about it, too.

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 2, 2016, 6:10pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/7 "2016-09-02T18:10:06Z")

</div>

i need to keep 1501 field in one index  
example of row data for import via fluentd

```
head -1 2016-09-01.csv
2016-09-01,121212,ATS,1.1.1.1, 0.0, 0.0, 518400.0, 0.0, 84.61, 88.71, 2266186.8200000008, 25920.0, 0.0, 7.0, 19.0, 680.0, 231982.0, 25920.0, 0.0, 32.72, 43.64, 1036976.8099999999, 25920.0, 0.0, 6.81, 0.0, 117110.0, 9.2340136E7, 281574.0, 172800.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.31, 0.0, 10263.0, 5.4668023E7, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 84.0, 117110.0, 8640.0, 0.0, 16.0, 10263.0, 8640.0, 1000.0, 0.0, 0.0, 0.0, 0.0, 83.1, 21.14, 8640.0, 8640.0, 1.1806565E7, 101250.0, 8.209024E7, 0.0, 0.0, 1.18539071E8, 101250.0, 7.0656471E7, 0.0, 0.0, 65383.0, 657481.0, 0.0, 18098.44, 0.0, 51515.62, 6650451.0, 6.3647587E7, 0.0, 7497565.0, 1.12485174E8, 397260.0, 0.0, 0.0, 0.0, 7.991577E8, 44749.0, 75331.0, 1305765.0, 466560.0, 466560.0, 0.0, 13392.0, 0.0, 46883.2, 0.0, 73264.0, 9.6827076E7, 5.3476308E7, 7197944.0, 0.0, 18080.0, 5.3341521E7, 1.11585122E8, 0.0, 56624.0, 9.1094716E7, 4665600.0, 4665600.0, 0.0, 0.0, 0.0, 0.0, 0.0
```

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 7, 2016, 6:51am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/8 "2016-09-07T06:51:48Z")

</div>

Now i know that 1501 fields in same index are no problem.  
But in my case the problem is scientific notation in incoming data .  
Elastic fully ignore document with even one scientific notation value, for example :

> ,0.0,0.0,0.0,1.1933174e+7,518400.0,0.0,1.1403393e+7,518400.0,0.0,19971.900000000005  
> all values are "type": "float" is it correct ?

i don not know how to avoid scientific notation  
?

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 7, 2016, 2:38pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/9 "2016-09-07T14:38:37Z")

</div>

maybe someone know how to check it ?

> ```
> "properties": {
> "@timestamp": {
> "type": "date",
> "format": "strict_date_optional_time||epoch_millis"
> },
> "first (count)": {
> "type": "long"
> }
> }
> 
> ```

> PUT /pm-1/ats/3  
> {  
> "@timestamp" : "2016-09-07",  
> "first (count)": " **2e+3"**  
> }

```
{
   "error": {
      "root_cause": [
         {
            "type": "mapper_parsing_exception",
            "reason": "failed to parse [first (count)]"
         }
      ],
      "type": "mapper_parsing_exception",
      "reason": "failed to parse [first (count)]",
      "caused_by": {
         "type": "number_format_exception",
         "reason": "For input string: \" **2e+3** \""
      }
   },
   "status": 400
}

```

somebody know the solution ?

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 8, 2016, 6:20am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/10 "2016-09-08T06:20:40Z")

</div>

in my understanding i have to prepare incoming values by using bash...  
i mean recalculate each value 2e+3 = 2000

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [September 8, 2016, 9:33am UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/11 "2016-09-08T09:33:44Z")

</div>

> [@dkspace](#):
>
> somebody know the solution ?

No quotes?

```
POST /test/doc
{ 
	"@timestamp" : "2016-09-07",
	"first (count)": 2e+3
}

```

---

<div class="post-metadata">

**Author:** ![dkspace](https://avatars.discourse-cdn.com/v4/letter/d/bc79bd/32.png) [@dkspace](https://discuss.elastic.co/u/dkspace)\
**Post date:** [September 8, 2016, 2:06pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/12 "2016-09-08T14:06:40Z")

</div>

Thank you! )) this quotes made my week . w/o quotes e\_notation working!  
FYI : but normal notation is working with quotes also....

> ```
> {   
> "@timestamp" : "2016-09-07",
> "first (count)": "9999"
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:21pm UTC](https://discuss.elastic.co/t/scientific-notation-in-incoming-data/59517/13 "2017-07-05T22:21:48Z")

</div>