Error : Limit of total fields [1000] in index has been exceeded

Elastic Stack Elasticsearch
1

Hi, there elastic community,
I know this problem has already been exposed and even solved in some cases, but the solutions provided don't solve mine. Or I don't know how to apply them to my case.

Tools:

  • Elasticsearch: 6.1
  • Logstash: 6.1.2
  • Kibana: 6.1
  • Postman

Here's what I've done in Postman:
Create the indexindex: index :
PUT http://localhost:9200/test_jira/timespent/test
with: test_jira = index, timespent=type, test= document

Increased total fields limit:
"index.mapping.total_fields.limit": 10000
I have tried at first with 2000 then 3000 and went to 10 000

Put some fields in the document:
PUT http://localhost:9200/test_jira/timespent/test
JSON in the body of Postman

It worked for 2 users and no more, but actually, I still have 94 other users to load and the error I get is:

                "reason": "Limit of total fields [1000] in index [test_jira] has been exceeded"

Is there anyone who could give a hand, please?

2

How many of the fields are actually useful to you for searching or aggregations?

I'm not familiar with Postman but it appears it is capable of spawning a lot of new fieldnames.
It sounds like it would make sense to define a stricter mapping in elasticsearch that indexes the fields it cares about and either errors or sets enabled:false in the dynamic field mappings for new fields outside the set known to be of interest.

3

What does GET put_index_name_here/_settings show you?
I'm not on ES 6 yet but had the same issue lately and increased the limit to 1500

This is what I see now

{
" put_index_name_here": {
"settings": {
"index": {
"mapping": {
"total_fields": {
"limit": "1500"
...

For some reason that will not format...

Anyway, you can't change that on an existing index (at least in ES5) so I create our daily indices in advance to set these settings. I hope to have a more elegant solution when jumping to ES6 but it kind of works for now :slight_smile:

4 
GET test_jira/_settings

gives:

{
  "test_jira": {
    "settings": {
      "index": {
        "creation_date": "1517563878815",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "6eCY7oklTa-4cyYwd6sppA",
        "version": {
          "created": "6010299"
        },
        "provided_name": "test_jira"
      }
    }
  }
}

Doesn't show anything about the field limits. However, the following command:

GET test_jira/_settings

Shows it:

{
  "_index": "test_jira",
  "_type": "timespent",
  "_id": "test",
  "_version": 19,
  "found": true,
  "_source": {
    "settings": {
      "index.mapping.total_fields.limit": 1500,
      "number_of_shards": 1,
      "number_of_replicas": 0
    },
+ the document loaded
1 Like
5

The GET commands look really similar :stuck_out_tongue:

GET test_jira

That should show you all fields. Maybe that will help figure out what's going on. Sounds like very verbose logging if you exhaust the fields limit after two users

The limit is per index so maybe have one index per user? Also doesn't really sound like a great solution but, you know...

6

Hi Mark,
Actually, Postman does exactly the same work as the terminal, it's just an interface.
Instead of doing:
curl -XPUT in the terminal, you select PUT action, give the URL and give the JSON in the body part...
I even do it in Kibana Dev Tools, it gives the same results.

7

Was thinking about it, or maybe at least a document per user... don't know how to handle them after that...

8

doesn't work, reached 7 documents (7 users) and got the same error :frowning:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.