Failed to put mappings on indices metricbeat-6.6.1-2019.11.15

I just performed a rolling update on our elastic cluster from 6.6.1 to 6.8 to 7.4 and now I am no longer creating new indices from metricbeat or auditbeat starting when I upgraded the beats clients from 6.8 to 7.4. The client side looks fine it is connecting to the server and reporting every 30 seconds as configured. And when I look at discover I see logs, what I am not seeing is new indices when looking at index management. I have one new index "metricbeat-7.4.2-2019.11.13-0000001" from the day I installed the updates, same for auditbeat and none since then.

I found the following error on the server side but in my googling I saw mostly things related to logstash. I am not using any custom index templates the index template was created by running metricbeat setup --index-management

[2019-11-15T17:47:26,206][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [elastic01] failed to put mappings on indices [[[metricbeat-6.6.1-2019.11.15/JjjvYxD8SpGFiva_cx3ULw]]], type [doc]
java.lang.IllegalArgumentException: Rejecting mapping update to [metricbeat-6.6.1-2019.11.15] as the final mapping would have more than 1 type: [_doc, doc]
	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest( ~[elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute( ~[elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks( ~[elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs( ~[elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks( [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.MasterService.access$000( [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.MasterService$ [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed( [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.cluster.service.TaskBatcher$ [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean( [elasticsearch-7.4.2.jar:7.4.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedR [elasticsearch-7.4.2.jar:7.4.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker( [?:?]
	at java.util.concurrent.ThreadPoolExecutor$ [?:?]
	at [?:?]

So turns out a few things are going on,

  1. Was a change in Index Lifecycle Policies. As of 7.0 indices are not created on a daily basis. The default policy is every 50 GB or 30 days, which ever happens first.
  2. I still have some old 6.6.1 clients in the environment that I need to update.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.