Hi,
We are planning use ElasticSearch in one of our projects. Currently, we are testing ElasticSearch 5.0.1 with our data. One issue we are facing is when we are doing a bulk upload from our MySQL tables to elasticsearch following error we are getting...
java.lang.IllegalArgumentException: Limit of total fields [1000] in
index [shopfront] has been exceeded
We are using PHP as elasticsearch client to doing the bulk upload from MySQL to Elastic. After doing some googling I got this piece of info - ES 2.3 -> 5.x metricbeat index field limit
Somewhere also I read that using of "index.mapping.total_fields.limit" will fix the thing. But, can't able to understand how to using that in my PHP code. Here is my PHP code.
$params = ['body' => []];
$i = 1;
foreach ($productsList as $key => $value) {
$params['body'][] = [
'index' => [
'_index' => 'shopfront',
'_type' => 'products'
],
'settings' => ['index.mapping.total_fields.limit' => 3000]
];
$params['body'][] = [
'product_displayname' => $value['product_displayname'],
'product_price' => $value['product_price'],
'popularity' => $value['popularity'],
'lowestcomp_price' => $value['lowestcomp_price']
];
// Every 1000 documents stop and send the bulk request
if ($i % 1000 == 0) {
$responses = $client->bulk($params);
// erase the old bulk request
$params = ['body' => []];
// unset the bulk response when you are done to save memory
unset($responses);
}
$i++;
}
// Send the last batch if it exists
if (!empty($params['body'])) {
$responses = $client->bulk($params);
}
NOTE - I've used same code with Elasticsearch 2.4.1 & it's working fine with that.