I am putting various kind of logs into Elasticsearch with daily time-based index (somedata-YYYY.MM.DD).
Recently, I have started to put many other kind of logs into ES, then ES started to logs many ProcessClusterEventTimeoutException after 00:00 AM.
[2016-10-19 00:00:30,858][DEBUG][action.admin.indices.mapping.put] [myhostname] failed to put mappings on indices [[somedata-2016.10.18]], type [fluentd]
ProcessClusterEventTimeoutException[failed to process cluster event (put-mapping [fluentd]) within 30s]
Actually, I have more than 200 indices every day, so when the date changes, almost all of them will need new index for the next day.
I suspected that some node (maybe master node) was under high CPU load, but as show below, CPU usage is under 5% in all nodes(servers), and CPU usage is dropped during this time period (00:00-00:10 AM)
It seems like ES is not under high load but something is blocking the operation. Does anyone suggest how can I investigate this?
By the way, I am running 10 nodes Elasticsearch cluster with version 2.4.0.