Elasticsearch bulk upload error with PHP - Limit of total fields [1000] in index has been exceeded

Elastic Stack Elasticsearch
1

Hi,
We are planning use ElasticSearch in one of our projects. Currently, we are testing ElasticSearch 5.0.1 with our data. One issue we are facing is when we are doing a bulk upload from our MySQL tables to elasticsearch following error we are getting...

java.lang.IllegalArgumentException: Limit of total fields [1000] in
index [shopfront] has been exceeded

We are using PHP as elasticsearch client to doing the bulk upload from MySQL to Elastic. After doing some googling I got this piece of info - ES 2.3 -> 5.x metricbeat index field limit

Somewhere also I read that using of "index.mapping.total_fields.limit" will fix the thing. But, can't able to understand how to using that in my PHP code. Here is my PHP code.

$params = ['body' => []];

$i = 1;
foreach ($productsList as $key => $value) {

    $params['body'][] = [
        'index' => [
            '_index' => 'shopfront',
            '_type' => 'products'
        ],
        'settings' => ['index.mapping.total_fields.limit' => 3000]
    ];

    $params['body'][] = [
        'product_displayname' => $value['product_displayname'],
        'product_price' => $value['product_price'],
        'popularity' => $value['popularity'],
        'lowestcomp_price' => $value['lowestcomp_price']
    ];

    // Every 1000 documents stop and send the bulk request
    if ($i % 1000 == 0) {
        $responses = $client->bulk($params);

        // erase the old bulk request
        $params = ['body' => []];

        // unset the bulk response when you are done to save memory
        unset($responses);
    }

    $i++;
}

// Send the last batch if it exists
if (!empty($params['body'])) {
    $responses = $client->bulk($params);
}

NOTE - I've used same code with Elasticsearch 2.4.1 & it's working fine with that.

2

This setting is to be added when you create the index, not when you index a document.

3

@dadoonet : I'd post the same question on StackOverflow also & manage to get the solution. But, still, I've some doubt. Can you help me to clear it out ?

Here is my Stackoverflow post : http://stackoverflow.com/questions/40857060/elasticsearch-bulk-upload-error-with-php-limit-of-total-fields-1000-in-index

Thanks for replying back.

4

There is a reason this has been introduced as a safeguard, and that is to prevent issues around mapping explosion. I would recommend looking into where all these fields come from to ensure it will not grow indefinitely. Increasing the limit temporarily resolve the error you are seeing, but could cause problems down the line.

1 Like
5

Ok Got one doubt. If I've 2 INDEX like...

  1. shopfront/product
  2. shopfront1/product

In this case, INDEX name is different but the INDEX's TYPE is same. By any chance, FIELDS of 'product' TYPE from INDEX 'shopfront' clash with FIELDS of 'product' TYPE of INDEX 'shopfront1' ?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.