Dynamic Mapping and Pending Tasks

(Anantha Govindarajan) #1


We are dynamically adding new fields to our index. It creates large number of pending tasks especially "refresh-mapping" tasks . We are creating daily indices , once a day rotated for the first 7 or 8 hours we have 4000 to 5000 refresh-mapping tasks on same index(and same type too). On looking into es source found the following commands in MetaDataMappingService.executeRefreshOrUpdate()

 * Batch method to apply all the queued refresh or update operations. The idea is to try and batch as much
 * as possible so we won't create the same index all the time for example for the updates on the same mapping
 * and generate a single cluster change event out of all of those.

For how many refresh-mapping tasks , a update-mapping task will be called or is there a time-limit? What metric "as much as possible" refers ? (Sorry if i missing anything ?)

Due to this large number of pending tasks , node joining tasks also getting delayed . Node-disconnects had IMMEDIATE priority where as Node-Joining has only HIGH priority. Due to this priority mismatch client nodes facing little delay to join the cluster .

What is the purpose of "indices.cluster.send_refresh_mapping" property ? What happen if i set indices.cluster.send_refresh_mapping to false in production (whether It is recommended )?

(Mark Walkom) #2

What version are you on?

(Anantha Govindarajan) #3

Thanks for the reply Mark. We are using elasticsearch 1.6.0.

(system) #4