# Limit of total fields \[num\] in index \[logstash-XX\] has been exceeded **URL:** \ **Category:** Logstash\ **Created:** [October 30, 2019, 2:56pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889 "2019-10-30T14:56:54Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![wpirraglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wpirraglia/32/56874_2.png) [@wpirraglia](https://discuss.elastic.co/u/wpirraglia)\ **Post date:** [October 30, 2019, 2:56pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/1 "2019-10-30T14:56:54Z")
Little background story first: We had an ELK 2.6 stack, which we decided to upgrade to 6.5. After creating the new stack, we decided to use the reindex API to copy our old Logstash indexes from the old stack to the new one, skipping the documents that threw an error. After some time, we started getting those errors at our Logstash logs, pointing to a limit in the number of fields. We increased the limit of the Logstash index template, and everything started working fine. But after a few weeks, we started getting those same errors, but now they were exceeding the new number of fields. Could this be related to changes in pipeline.batch.size and pipeline.batch.dealy parameters in logstash.yml (we did some tuning of our logstash processes)? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 30, 2019, 3:13pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/2 "2019-10-30T15:13:27Z")
> [@wpirraglia](#): > > Could this be related to changes in pipeline.batch.size and pipeline.batch.dealy parameters No, those would not result in additional fields being created. ---
**Author:** ![wpirraglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wpirraglia/32/56874_2.png) [@wpirraglia](https://discuss.elastic.co/u/wpirraglia)\ **Post date:** [October 30, 2019, 4:07pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/3 "2019-10-30T16:07:04Z")
I know, and maybe it was a coincidence, but when I changed those values, the error started happening. Adding here some of the errors I'm getting: ` [2019-10-30T14:13:19,567][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.10.30", :_type=>"doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"logstash-2019.10.30", "_type"=>"doc", "_id"=>"XnEDHW4BAwjYVExVfrLz", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"Limit of total fields [3000] in index [logstash-2019.10.30] has been exceeded"}}}}` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 30, 2019, 4:25pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/4 "2019-10-30T16:25:58Z")
> [@wpirraglia](#): > > failed to parse field [respons e.body.id] Do you really have spaces in your field names? In Kibana you can see a list of the field names in the index. If I recall correctly it is under index management. Does that list look reasonable to you? ---
**Author:** ![wpirraglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wpirraglia/32/56874_2.png) [@wpirraglia](https://discuss.elastic.co/u/wpirraglia)\ **Post date:** [October 30, 2019, 4:39pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/5 "2019-10-30T16:39:18Z")
> [@Badger](#): > > Do you really have spaces in your field names? Not really, that's related to a mapping error I'm also fixing (removed it from the error I pasted earlier). Is there a way to check the document that it's supposed to have more than X fields (3000 in this case)? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 30, 2019, 5:02pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/6 "2019-10-30T17:02:11Z")
> [@wpirraglia](#): > > Is there a way to check the document that it's supposed to have more than X fields (3000 in this case)? It is not saying that any single document has that many fields. It is saying that there are 3000 members in the set of fields that occur across all documents that have been indexed. It could be 3000 documents, each of which has a unique field name. That's why I am suggesting you review the list in Kibana. ---
**Author:** ![wpirraglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wpirraglia/32/56874_2.png) [@wpirraglia](https://discuss.elastic.co/u/wpirraglia)\ **Post date:** [October 30, 2019, 5:51pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/7 "2019-10-30T17:51:12Z")
Considering all our apps report to that ELK, I checked the amount of fields in the logstash indices and saw they increased over time. I need to check why, but that seems to be the cause of this particular problem. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 27, 2019, 5:51pm UTC](https://discuss.elastic.co/t/limit-of-total-fields-num-in-index-logstash-xx-has-been-exceeded/205889/8 "2019-11-27T17:51:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.