Update mapping is throttled while update index templates?

(jianxiong) #1


when I update a lot of index templates, there are some data indexed into a new Index Index-a,so Index-a need to update mapping, but from the master node log, I found a lot of log like:

[2019-04-17T17:32:48,271][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [jssz-billions-es-01-masternode] failed to put mappings on indices [[[billions-bplus-raw_log-http-proxy-@2019.04.17-jssz01-0/rFFmavUZQjC40AU2dLwu0A]]], type [logs]
org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s
[2019-04-17T17:32:48,283][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [jssz-billions-es-01-masternode] failed to put mappings on indices [[[billions-bplus-raw_log-http-proxy-@2019.04.17-jssz01-0/rFFmavUZQjC40AU2dLwu0A]]], type [logs]

when I shut the task of update templates, the mapping is successed updating.

So, update mapping will be throttled while update index templates?

(David Turner) #2

There's no throttling here, but both templates and mappings are managed by the master so they are updated in sequence, and these updates will fail if they don't execute fast enough. Can you tell us a bit more about your cluster: how many nodes, and how many shards does it have? Are you using dedicated master nodes? How much data are you indexing and how often are you searching it?

(jianxiong) #3

the cluster info is:

and the three master nodes are dedicated.

(David Turner) #4

Thanks. I think you have too many shards, so the master is under too much pressure to keep up. You have an average of over 1200 shards per node, and the average shard size is under 400MB. Here is an article with some more information on how many shards you should have:

(jianxiong) #5

Ok, thanks! I will read it and optimising our cluster.:grinning:

(jianxiong) #6

There is a new question for me, if the indexs are closed, will the amount of meta-information data of their shards decrease in the global cluster meta data?

(David Turner) #7

It's best to ask new questions on a new thread. Would you do that please?