Recently upgrade my elasticsearch to version 6.7.2 (along with upgrading graylog to 3.0.2). Now whenever rotate my active write index my elasticsearch cluster goes to red (this is a VM and so I have since reverted to before the rotate). Running "curl -XGET :9200/_cluster/allocation/explain?pretty" gives me an explanation of "no allocations are allowed due to cluster setting [cluster.routing.allocation.enable=none]".
I am not able to figure out from my elasticsearch log (below) what the problem could be. Could anyone be of any help here?
[2019-05-20T11:58:17,181][DEBUG][o.e.a.b.TransportShardBulkAction] [bwFrRi0] [graylog_74][3] failed to execute bulk item (index) index {[graylog_deflector][message][8d8f6801-7af6-11e9-b59b-0050568daa06], source[{"Task":0,"Keywords":4611686018429487122,"MachineEnvironment":"production","EventType":"ERROR","gl2_remote_ip":"","gl2_remote_port":54780,"Opcode":"Info","source":"computer.domain","gl2_source_input":"5877c7f3b85fe8038960fafa","SeverityValue":4,"Version":0,"UserID":"S-1-5-18","gl2_source_node":"75678ef0-d302-48d8-8b5e-f9a439b9e7d1","ProcessID":5836,"timestamp":"2019-05-20 11:58:14.000","OpcodeValue":0,"SourceModuleType":"im_msvistalog","level":3,"Channel":"Microsoft-Windows-LiveId/Operational","streams":["000000000000000000000001"],"SourceName":"Microsoft-Windows-LiveId","Severity":"ERROR","message":"SOAP Request of type Service for user CID 'NULL' in production e","AccountType":"User","EventReceivedTime":"2019-05-20 07:58:16","SourceModuleName":"in","ProviderGuid":"{05F02597-FE85-4E67-8542-69567AB8FD4F}","full_message":"SOAP Request of type Service for user CID 'NULL' in production environment received the following error code from the Microsoft Account server: 0x80041F0D.","ThreadID":18752,"EventID":6114,"ErrorCode":"2147753741","Domain":"NT AUTHORITY","RecordNumber":7407,"AccountName":"SYSTEM","RequestType":"1","cid":"NULL"}]}
java.lang.IllegalArgumentException: Limit of total fields [1000] in index [graylog_74] has been exceeded
at org.elasticsearch.index.mapper.MapperService.checkTotalFieldsLimit(MapperService.java:639) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:520) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:403) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:338) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:330) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:231) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:270) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:200) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:135) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215) ~[elasticsearch-6.7.2.jar:6.7.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
[2019-05-20T12:12:08,508][INFO ][o.e.c.m.MetaDataIndexTemplateService] [bwFrRi0] adding template [graylog-internal] for index patterns [graylog_*]
[2019-05-20T12:12:08,596][INFO ][o.e.c.m.MetaDataCreateIndexService] [bwFrRi0] [graylog_75] creating index, cause [api], templates [graylog-internal], shards [4]/[0], mappings [message]
[2019-05-20T12:12:08,611][INFO ][o.e.c.r.a.AllocationService] [bwFrRi0] Cluster health status changed from [YELLOW] to [RED] (reason: [index [graylog_75] created]).