Limit of total fields [1000] exceeded

I am getting Limit of total fields [1000] in index [products] has been exceeded. Although total number of properties that is added dynamically is 499. Every where I search is asking me to increase total_fields length since my limit haven't increased I am not planning to that. Following is the mapping of index

  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "bvid": {
        "type": "keyword"
      },
      "gtin": {
        "type": "keyword"
      },
      "manufacturerCode": {
        "type": "keyword"
      },
      "productState": {
        "properties": {
          "imageReady": {
            "type": "keyword",
            "index": false
          },
          "contentReady": {
            "type": "keyword",
            "index": false
          },
          "reviewReady": {
            "type": "keyword",
            "index": false
          }
        }
      },
      "physicalStatus": {
        "properties": {
          "status": {
            "type": "keyword",
            "index": false
          }
        }
      },

      "images": {
        "properties": {
          "url": {
            "type": "keyword",
            "index": false
          },
          "thumbnailUrl": {
            "type": "keyword",
            "index": false
          },
          "imageType": {
            "type": "keyword",
            "index": false
          },
          "isHero": {
            "type": "boolean",
            "index": false
          }
        }
      }
    }
  }
}

Rest are dynamically added.

It means that the data which is sent to your index contains too many new fields.
Could you share one document which is failing?

This is the document which is failing

{
      "name": "Good Goodies",
      "bvid": "c7befa56",
      "gtin": "8964002148134",
      "manufacturerCode": "209975",
      "images": [
        {
          "url": "https://storage.googleapis.com/",
          "thumbnailUrl": "https://storage.googleapis.com/\",
          "isHero": true,
          "imageType": "hero"
        }
      ],
      "productState": {
        "contentReady": "Complete",
        "imageReady": "Complete",
        "reviewReady": "None"
      },
      "physicalStatus": {
        "status": "Cleared"
      },
      "In Package Depth": "145",
      "Sodium Per Nutrient Basis Quantity": "135.0mg",
      "Packaging Type": "Box",
      "Level of Fat Claim": "UNIDENTIFIED",
      "Nutrient Basis Quantity": "100g",
      "Protein Per Nutrient Basis Quantity": "5.0g",
      "Type of Cake": "UNIDENTIFIED",
      "Packaging Shape": "Rectangular",
      "Sugars Per Nutrient Basis Quantity": "31.9g",
      "Packaging Recycling Process Type": "Recyclable",
      "GPC Category Code": "10000172",
      "Fibre Per Nutrient Basis Quantity": "0.4g",
      "Consumer Usage Instructions": "Store in room temperature.",
      "Net Weight": "252",
      "Ingredient Statement": "Wheat Flour,Sugar, Vegetable Oil,Invert Syrup, Fresh Eggs, Skimmed Milk Powder, Cocoa Powder, Salt, Baking Powder, Emulsifier, Potassium Sorbate, Citric Acid, Glycerin, Sorbitol, Vanillin Powder, Corn Starch, Food Preservatives, Xanthan Gum, Nature Identical Flavor and Permitted Food Color.",
      "Fat Per Nutrient Basis Quantity": "17.9g",
      "In Package Height": "70",
      "Nutrient Basis Quantity Type Code": "By Serving",
      "Product Range": "Cake",
      "Saturated Fat Per Nutrient Basis Quantity": "0.4g",
      "Special Occasion Claim": "NO",
      "Product Description": "Enjoy the three in one baked freshness of sweet fillings, great taste & nutrition.",
      "Gluten Free Claim": "NO",
      "Functional Name": "Good Goodies Merry Berry",
      "Product Form Description": "Box",
      "Packaging Feature": "Wrap",
      "Family": "Bread/Bakery Products",
      "Gross Weight": "300",
      "Energy from Fat Per Nutrient Basis Quantity": "399Cal",
      "Price Value": "108",
      "In Package Width": "180",
      "GTIN": "8964002148134",
      "Nutrient Measurement Precision": "Approximately",
      "Product Name": "Good Goodies Triple Strawberry Jam Filled Merry Berry 6 Long Cakes",
      "If Organic": "NO",
      "Cholesterol Per Nutrient Basis Quantity": "60.0mg",
      "Brand Name": "Good Goodies",
      "Carbohydrates Per Nutrient Basis Quantity": "54.5g",
      "Variant Description": "Merry Berry",
      "Class": "Sweet Bakery Products",
      "If Coated": "YES",
      "Packaging Function": "Protected",
      "Ingredient of Concern": "Contains Wheat, Eggs & Milk.",
      "GPC Category Description": "Cakes - Sweet (Shelf Stable)",
      "Brick": "Cakes - Sweet (Shelf Stable)",
      "Segment": "Food/Beverage/Tobacco",
      "If Part Baked": "YES",
      "Regulated Product Name": "Good Goodies Triple Strawberry Jam Filled Merry Berry",
      "Form of Cake": "UNCLASSIFIED",
      "Packaging Material Type": "Paper Paperboard",
      "Sub Brand": "NotApplicable",
      "Brand": "Good Goodies",
      "Brand Owner": "NoOwner"
    }

As you can see there are not 1000 fields

What is the current mapping?

This is my current mapping https://ctxt.io/2/AAAghxsPFA I can't paste it here as limit exceeded. They are total 499

I can reproduce the error.

If I look a the mapping file, you have around 4990 lines of mapping content.
Knowing that a field is approximately 5 lines, I'm pretty sure you are super close to 1000 fields.

1 Like

Oh I see what you did not understand...

When you have this for the mapping:

      "Additional Product Description": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }

It's actually 2 fields in elasticsearch: Additional Product Description and Additional Product Description.keyword.

1 Like

This explains a lot. Thanks a lot :slightly_smiling_face:

I'd suggest that you use a more flexible mapping.

Using may be a data structure like:

DELETE test
PUT test
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "bvid": {
        "type": "keyword"
      },
      "gtin": {
        "type": "keyword"
      },
      "manufacturerCode": {
        "type": "keyword"
      },
      "productState": {
        "properties": {
          "imageReady": {
            "type": "keyword",
            "index": false
          },
          "contentReady": {
            "type": "keyword",
            "index": false
          },
          "reviewReady": {
            "type": "keyword",
            "index": false
          }
        }
      },
      "physicalStatus": {
        "properties": {
          "status": {
            "type": "keyword",
            "index": false
          }
        }
      },
      "images": {
        "properties": {
          "url": {
            "type": "keyword",
            "index": false
          },
          "thumbnailUrl": {
            "type": "keyword",
            "index": false
          },
          "imageType": {
            "type": "keyword",
            "index": false
          },
          "isHero": {
            "type": "boolean",
            "index": false
          }
        }
      },
      "data": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "keyword"
          },
          "value": {
            "type": "text"
          }
        }
      }
    }
  }
}
POST test/_doc
{
  "name": "Good Goodies",
  "bvid": "c7befa56",
  "gtin": "8964002148134",
  "manufacturerCode": "209975",
  "images": [
    {
      "url": "https://storage.googleapis.com/",
      "thumbnailUrl": "https://storage.googleapis.com/",
      "isHero": true,
      "imageType": "hero"
    }
  ],
  "productState": {
    "contentReady": "Complete",
    "imageReady": "Complete",
    "reviewReady": "None"
  },
  "physicalStatus": {
    "status": "Cleared"
  },
  "data": [
  { "name": "In Package Depth", "value": "145" },
  { "name": "Sodium Per Nutrient Basis Quantity", "value": "135.0mg" },
  { "name": "Packaging Type", "value": "Box" },
  { "name": "Level of Fat Claim", "value": "UNIDENTIFIED" },
  { "name": "Nutrient Basis Quantity", "value": "100g" },
  { "name": "Protein Per Nutrient Basis Quantity", "value": "5.0g" },
  { "name": "Type of Cake", "value": "UNIDENTIFIED" },
  { "name": "Packaging Shape", "value": "Rectangular" },
  { "name": "Sugars Per Nutrient Basis Quantity", "value": "31.9g" },
  { "name": "Packaging Recycling Process Type", "value": "Recyclable" },
  { "name": "GPC Category Code", "value": "10000172" },
  { "name": "Fibre Per Nutrient Basis Quantity", "value": "0.4g" },
  { "name": "Consumer Usage Instructions", "value": "Store in room temperature." },
  { "name": "Net Weight", "value": "252" },
  { "name": "Ingredient Statement", "value": "Wheat Flour,Sugar, Vegetable Oil,Invert Syrup, Fresh Eggs, Skimmed Milk Powder, Cocoa Powder, Salt, Baking Powder, Emulsifier, Potassium Sorbate, Citric Acid, Glycerin, Sorbitol, Vanillin Powder, Corn Starch, Food Preservatives, Xanthan Gum, Nature Identical Flavor and Permitted Food Color." },
  { "name": "Fat Per Nutrient Basis Quantity", "value": "17.9g" },
  { "name": "In Package Height", "value": "70" },
  { "name": "Nutrient Basis Quantity Type Code", "value": "By Serving" },
  { "name": "Product Range", "value": "Cake" },
  { "name": "Saturated Fat Per Nutrient Basis Quantity", "value": "0.4g" },
  { "name": "Special Occasion Claim", "value": "NO" },
  { "name": "Product Description", "value": "Enjoy the three in one baked freshness of sweet fillings, great taste & nutrition." },
  { "name": "Gluten Free Claim", "value": "NO" },
  { "name": "Functional Name", "value": "Good Goodies Merry Berry" },
  { "name": "Product Form Description", "value": "Box" },
  { "name": "Packaging Feature", "value": "Wrap" },
  { "name": "Family", "value": "Bread/Bakery Products" },
  { "name": "Gross Weight", "value": "300" },
  { "name": "Energy from Fat Per Nutrient Basis Quantity", "value": "399Cal" },
  { "name": "Price Value", "value": "108" },
  { "name": "In Package Width", "value": "180" },
  { "name": "GTIN", "value": "8964002148134" },
  { "name": "Nutrient Measurement Precision", "value": "Approximately" },
  { "name": "Product Name", "value": "Good Goodies Triple Strawberry Jam Filled Merry Berry 6 Long Cakes" },
  { "name": "If Organic", "value": "NO" },
  { "name": "Cholesterol Per Nutrient Basis Quantity", "value": "60.0mg" },
  { "name": "Brand Name", "value": "Good Goodies" },
  { "name": "Carbohydrates Per Nutrient Basis Quantity", "value": "54.5g" },
  { "name": "Variant Description", "value": "Merry Berry" },
  { "name": "Class", "value": "Sweet Bakery Products" },
  { "name": "If Coated", "value": "YES" },
  { "name": "Packaging Function", "value": "Protected" },
  { "name": "Ingredient of Concern", "value": "Contains Wheat, Eggs & Milk." },
  { "name": "GPC Category Description", "value": "Cakes - Sweet (Shelf Stable)" },
  { "name": "Brick", "value": "Cakes - Sweet (Shelf Stable)" },
  { "name": "Segment", "value": "Food/Beverage/Tobacco" },
  { "name": "If Part Baked", "value": "YES" },
  { "name": "Regulated Product Name", "value": "Good Goodies Triple Strawberry Jam Filled Merry Berry" },
  { "name": "Form of Cake", "value": "UNCLASSIFIED" },
  { "name": "Packaging Material Type", "value": "Paper Paperboard" },
  { "name": "Sub Brand", "value": "NotApplicable" },
  { "name": "Brand", "value": "Good Goodies" },
  { "name": "Brand Owner", "value": "NoOwner" }
  ]
}

This is much more flexible and is keeping the mapping as minimal as possible.
It comes with more complex queries though as you need to deal with nested objects. See

I was doing that previously but I wasn't able to do some searches where I want to make multiple name and value filter ,I ended up doing this:

{
  "mappings": {
    "dynamic_templates": [
      {
        "strings": {
          "match_mapping_type": "string",
          "mapping": {
            "type": "text"
          }
        }
      }
    ],
    "properties": {
      "name": {
        "type": "text"
      },
      "bvid": {
        "type": "keyword"
      },
      "gtin": {
        "type": "text"
      },
      "manufacturerCode": {
        "type": "text"
      },
      "productState": {
        "properties": {
          "imageReady": {
            "type": "keyword",
            "index": false
          },
          "contentReady": {
            "type": "keyword",
            "index": false
          },
          "reviewReady": {
            "type": "keyword",
            "index": false
          }
        }
      },
      "physicalStatus": {
        "properties": {
          "status": {
            "type": "keyword",
            "index": false
          }
        }
      },

      "images": {
        "properties": {
          "url": {
            "type": "keyword",
            "index": false
          },
          "thumbnailUrl": {
            "type": "keyword",
            "index": false
          },
          "imageType": {
            "type": "keyword",
            "index": false
          },
          "isHero": {
            "type": "boolean",
            "index": false
          }
        }
      }
    }
  }
}

Then explain what your problem was and we might help you solving the query part.

But IMO don't use a bad design because you don't know how to query...

My query was I wanted to search on like on both of these value

[{ "name": "In Package Depth", "value": "145" },
  { "name": "Sodium Per Nutrient Basis Quantity", "value": "135.0mg" }]

I wrote a query in which I can search on both conditions but I wasn't able too. I was able to search when I apply one condition only. My query is like this

{
    "query": {
        "bool": {
            "must": [
                {
                    "nested": {
                        "path": "attributes",
                        "query": {
                            "bool": {
                                "must": [
                                    {
                                        "match": {
                                            "attributes.name": "In Package Depth"
                                        }
                                    },
                                    {
                                        "match": {
                                            "attributes.value": "135"
                                        }
                                    },
                                    {
                                        "match": {
                                            "attributes.name": "Sodium Per Nutrient Basis Quantity"
                                        }
                                    },
                                    {
                                        "match": {
                                            "attributes.value": "145"
                                        }
                                    }
                                ]
                            }
                        }
                    }
                }
            ]
        }
    }
}

You need to write multiple nested queries in that case. Like this:

DELETE test
PUT test
{
  "mappings": {
    "properties": {
      "attributes": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "keyword"
          },
          "value": {
            "type": "text"
          }
        }
      }
    }
  }
}
POST test/_doc
{
  "attributes": [
  { "name": "In Package Depth", "value": "145" },
  { "name": "Sodium Per Nutrient Basis Quantity", "value": "135.0mg" }
  ]
}

GET test/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "nested": {
            "path": "attributes",
            "query": {
              "bool": {
                "must": [
                  {
                    "match": {
                      "attributes.name": "In Package Depth"
                    }
                  },
                  {
                    "match": {
                      "attributes.value": "145"
                    }
                  }
                ]
              }
            }
          }
        },        
        {
          "nested": {
            "path": "attributes",
            "query": {
              "bool": {
                "must": [
                  {
                    "match": {
                      "attributes.name": "Sodium Per Nutrient Basis Quantity"
                    }
                  },
                  {
                    "match": {
                      "attributes.value": "135.0mg"
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  }
}

Please also note that 135 does not match 135.0mg, because of the default analyzer:

POST test/_analyze
{
  "analyzer": "default",
  "text": ["135.0mg"]
}
POST test/_analyze
{
  "analyzer": "default",
  "text": ["135"]
}

Gives:

# POST test/_analyze
{
  "tokens" : [
    {
      "token" : "135.0mg",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "<ALPHANUM>",
      "position" : 0
    }
  ]
}

# POST test/_analyze
{
  "tokens" : [
    {
      "token" : "135",
      "start_offset" : 0,
      "end_offset" : 3,
      "type" : "<NUM>",
      "position" : 0
    }
  ]
}

Do I need to change the analyzer too I didn't get this part. Thanks a lot I will try it like too. I am using this mapping in my other storage for ES where I am storing limited data but here I need to add whole data for searching. Thank you again for helping me with the query

Yes if you need that 135 actually matches things like 135, 135.0, 135.0g, 135.0mg.
IMHO it's better to solve this at index time by sending data like:

POST test/_doc
{
  "attributes_text": [
    { "name": "Packaging Type", "value": "Box" }
  ],
  "attributes_int": [
    { "name": "In Package Depth", "value": 145 }
  ],
  "attributes_float": [
    { "name": "Sodium Per Nutrient Basis Quantity in mg", "value": 135.0 }
  ]
}

That way you can also think of using range queries on numerical fields.

And there will be separate properties for "attributes_text" , "attributes_int" and "attributes_float"?

You can do multiple things.

One is:

DELETE test 
PUT test
{
  "mappings": {
    "properties": {
      "attributes_text": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "keyword"
          },
          "value": {
            "type": "text"
          }
        }
      },
      "attributes_int": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "keyword"
          },
          "value": {
            "type": "integer"
          }
        }
      }
    }
  }
}

Another idea which will require less fields:

DELETE test 
PUT test
{
  "mappings": {
    "properties": {
      "attributes": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "keyword"
          },
          "value_text": {
            "type": "text"
          },
          "value_int": {
            "type": "integer"
          }
        }
      }
    }
  }
}

Thanks a lot for your help :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.