# Mapping http\_poller JSON to an index

**URL:** <https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756>\
**Category:** Elasticsearch\
**Created:** [May 23, 2017, 5:01am UTC](https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756 "2017-05-23T05:01:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![digs\_77](https://avatars.discourse-cdn.com/v4/letter/d/eada6e/32.png) [@digs\_77](https://discuss.elastic.co/u/digs_77)\
**Post date:** [May 23, 2017, 5:01am UTC](https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756/1 "2017-05-23T05:01:17Z")

</div>

This is a representation of a JSON array (widgets) I'm getting with the http\_poller plugin...

```
 {
 "ABC_123":
      {"id": 7, "cost": 1.02, "sales_volume": 6}, 
 "3BT_8RT":
      {"id": 84, "cost": .025, "sales_volume": 7800}
 }

```

There are about 90 items (widgets) in the JSON array  
ID's don't change, and there are actually 11 fields... I shortened for brevity.  
I also need to index the names (ABC\_123) for each item, they're unique and also stay with the ID property.  
Lastly, I want a datetime on every widget entry.

My PUT mapping...

```
PUT /widgetdata
{
  "mappings": {
    "widgets" : {
      "properties" : {
        "id" : {"type" : "keyword"},
        "cost" : {"type" : "float"},
        "sales_volume" : {"type" : "float"},
        "datetime" : {"type" : "date", "format": "yyyyMMdd'T'HHmmss.SSSZ"}
       }
    }
  }
}

```

And finally, my logstash config file...

```
input {
    http_poller {
        urls => {
          ticker => "https://widgetsite.com/public?command=returnWidgets"
        }
        request_timeout => 60
        # Supports "cron", "every", "at" and "in" schedules by rufus scheduler
        schedule => { every => "5s" }
        codec => "json"
        # A hash of request metadata info (timing, response headers, etc.) will be sent here
        metadata_target => "http_poller_metadata"
     }
}
filter {
  grok { match => ["message", "%{HTTPDATE:[@metadata][timestamp]}" ] }
  date { match => ["[@metadata][timestamp]", "yyyyMMdd'T'HHmmss.SSSZ" ] }
}
output {
    stdout {
        codec => "rubydebug"
    }
    elasticsearch {
        hosts => "localhost"
        index => "widgets-%{+YYYY.MM.dd}"
        user => "elastic"
        password => " ********"
    }
}

```

The error from rubydebug says I'm exceeding the 1000 field limit.

`04:52:39.324 [[main]>worker3] WARN logstash.outputs.elasticsearch - Failed action. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"widgets-2017.05.23", :_type=>"logs", :_routing=>nil}, 2017-05-23T04:52:38.707Z %{host} %{message}], :response=>{"index"=>{"_index"=>"widgets-2017.05.23", "_type"=>"logs", "_id"=>"AVwzplqg3CbsVytXtm70", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [1000] in index [widgets-2017.05.23] has been exceeded"}}}}`

Not sure why it won't use my ID, it's assigning a new ID it appears?

My guess is that my mapping is incorrect and diving through the mapping api docs, I can't figure out where I'm going wrong. I say guess, because I have 90 items in the JSON and 11 fields per. That would be over 1000 if Elastic or Logstash is trying to create a new field for every item parameter in the array. And would makes sense why I get this error. But very unsure how to fix...???

Thanks for any help or suggestions!

---

<div class="post-metadata">

**Author:** ![Jai\_Prakash\_Bhardwaj](https://avatars.discourse-cdn.com/v4/letter/j/49beb7/32.png) [@Jai\_Prakash\_Bhardwaj](https://discuss.elastic.co/u/Jai_Prakash_Bhardwaj)\
**Post date:** [May 23, 2017, 6:23am UTC](https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756/2 "2017-05-23T06:23:57Z")

</div>

Hi,

> [@digs\_77](#):
>
> elasticsearch {  
> hosts =\> "localhost"  
> index =\> "widgets-%{+YYYY.MM.dd}"  
> user =\> "elastic"  
> password =\> "\*\*\*\*\*\*\*\*"  
> }  
> }

you need to add

```auto
document_id => "%{_id}" 

```

to your elasticsearch output plugin to make the Id as the elasticsearch id.

Hope this helps

Best,  
J

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 23, 2017, 6:51am UTC](https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756/3 "2017-05-23T06:51:25Z")

</div>

> [@digs\_77](#):
>
> This is a representation of a JSON array

That looks more like a single JSON object where the ID is used as a key than an array. If you look at the stdout output, is this being indexed as a single and quite complex document?

---

<div class="post-metadata">

**Author:** ![digs\_77](https://avatars.discourse-cdn.com/v4/letter/d/eada6e/32.png) [@digs\_77](https://discuss.elastic.co/u/digs_77)\
**Post date:** [May 23, 2017, 6:52am UTC](https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756/4 "2017-05-23T06:52:23Z")

</div>

Thanks for the suggestion, I changed it to this...

```
output {
    stdout {
        codec => "rubydebug"
     }
     elasticsearch {
         hosts => "localhost"
         index => "widgets-%{+YYYY.MM.dd}"
         user => "elastic"
         password => " *********"
         document_id => "%{_id}"
     }
 }

```

But get the same "Limit of total fields [1000] in index..." error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2017, 6:52am UTC](https://discuss.elastic.co/t/mapping-http-poller-json-to-an-index/86756/5 "2017-06-20T06:52:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.