How can I set the field limit higher then 1000?

Elastic Stack Elasticsearch
1

I am new to Elastic Search (visited the Elastic Event in Frankfurt today, thanks for the great talk and insights btw).

I get this error when indexing products from shopware:

[2017-09-28T22:58:13,583][DEBUG][o.e.a.b.TransportShardBulkAction] 
[fE76H5K] [sw_shop5_20170928225616][3] failed to execute bulk item 
(index) BulkShardRequest [[sw_shop5_20170928225616][3]] containing 
[index {[sw_shop5_20170928225616][product][A40482001], source[n/a, 
actual length: [41.6kb], max length: 2kb]}]

and

Limit of total fields [1000] in index [sw_shop5_20170928233439] has been exceeded

I generated a template to set the field limit higher:

curl -XGET 'localhost:9200/_template?pretty'
{
  "shopwaretemplate" : {
    "order" : 0,
    "template" : "shopwaretemplate",
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "100000"
          }
        }
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

but it still throws the error and I dont know how to set the 2kb higher, so I can import my data. Sorry if my terminology is not good yet.

2

Did you remove the index after creating the template?
Could you check what the actual index settings are?

3

Yes, the software does that any time I populate it (and keeps the old index). Tho I am not sure how to get the template of an index

http://localhost:9200/sw_shop5_20170928233439/_template

Doesn`t do the trick.

By the way is this:

[2017-09-29T07:57:15,683][DEBUG][o.e.a.b.TransportShardBulkAction] [fE76H5K] [sw_shop5_20170929075533][3] failed to execute bulk item (index) BulkShardRequest [[sw_shop5_20170929075533][3]] containing [index {[sw_shop5_20170929075533][product][A40482001], source[n/a, actual length: [41.6kb], max length: 2kb]}]

Part of the problem or annother problem/config thing?

4

How are you ingesting data? You seem to have an index name with a very granular timestamp. Is this automatically generated?

5

Hi Christian,
yes, it is. Software which generates this index is Shopware - I heard from annother Dev that in case of multiple price groups and shops it will pass the 1000 fields quickly.

Question is - change the max field value (might lose performance) or change the exported data. If there is is a way anyways to extend the max fields value before indexing starts.

6

I think this section of the docs describe what you are looking for.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.