Issues upgrading Elasticsearch from 8 to 9 due to old 7.x indices and restricted Kibana indices

Elastic Stack Elasticsearch
1

Hello,

I’m currently upgrading our Elasticsearch cluster from version 8 to 9. During the process, I still had some old indices created with version 7.

Using the Upgrade Assistant, I marked all the old 7.x indices as read-only via the API by setting the verified_read_only flag. After that, I proceeded with the upgrade.

However, I ran into the initial error:

java.lang.IllegalStateException: The index [.kibana_7.17.21_001/...]
created in version [7.17.21] with current compatibility version [7.17.21]
must be marked as read-only using the setting [index.blocks.write] set to [true]
before upgrading to 9.0.2.

The main problem was that the .kibana_7.x... system index did not appear in the Upgrade Assistant, so I wasn’t aware it also needed the verified_read_only setting applied before upgrading.

Another challenge I’m facing is that when I try to set the .kibana_7.x index to verified_read_only manually, I get this error:

"action [indices:admin/settings/update] is unauthorized for user []
with effective roles [superuser,test] on restricted indices [.kibana_7.17.10_001],
this action is granted by the index privileges [manage,all]"

So despite having superuser role, I’m blocked by Elasticsearch’s restricted indices protection, which requires explicit manage or all privileges on these system indices.

If anyone has tips on:

  • How to discover all indices that still require the verified_read_only flag before upgrading,
  • Or how to handle the permission issues on restricted system indices during this process,

that would be a huge help!

Thanks in advance.

2

Hello @axvfvv79zcx57xv7k

Please review below post where similar issue reported during upgrade :

Thanks!!

3

In the Upgrade Assistant there is a box to migrate system indices. You need to actually do that, it's fairly easy to miss (IMO).

Yep, thats a dead end really.

There's a command (well, almost a script) to look at the version of all your indices (other ways too) in the linked post that @Tortoise shared.

You need to be upgrading via 8.18.x btw, also fairly easy to not know this.

Good luck, let us know how you get on.

4

Thanks for these usefull informations. I am now getting the following error

System indices migration failed

An error ocurred while migrating system indices for async_search: `illegal_state_exception`

Is there any way to get more information, why this is happening?

5

what does a GET on

/.async-search/_settings

return?

And a GET on

/_cat/indices/.async-search?v

and while we are at it, what about current output from:

curl -sk -u "${EUSER}:${EPASS}" "https://${EHOST}:${EPORT}" --request-target '_all/_settings?expand_wildcards=all' -X GET | jq -r 'to_entries[] | "\(.value.settings.index.creation_date) \(.value.settings.index.version.created) \(.key)"' | sort -k1nr -k2nr | while read f1 f2 f3 ; do echo $f2 $f1 $(date --utc --iso=seconds -d @$(( $f1  / 1000 )) ) $f3 ; done

(obfuscate your index names if you feel you need to)

6

/.async-search/_settings

brings

#! this request accesses system indices: [.async-search], but in a future major version, direct access to system indices will be prevented by default
{
  ".async-search": {
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        },
        "hidden": "true",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": ".async-search",
        "creation_date": "1589908315713",
        "number_of_replicas": "0",
        "uuid": "gixm9w2_TSyWf0UuNrAxSA",
        "version": {
          "created": "7070099",
          "upgraded": "7110299"
        }
      }
    }
  }
}

and GET /_cat/indices/.async-search?v
brings

health status index         uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
green  open   .async-search gixm9w2_TSyWf0UuNrAxSA   1   0          0            0       266b           266b         266b

curl -sk -u "${EUSER}:${EPASS}" "https://${EHOST}:${EPORT}" --request-target '_all/_settings?expand_wildcards=all' -X GET | jq -r 'to_entries[] | "\(.value.settings.index.creation_date) \(.value.settings.index.version.created) \(.key)"' | sort -k1nr -k2nr | while read f1 f2 f3 ; do echo $f2 $f1 $(date --utc --iso=seconds -d @$(( $f1  / 1000 )) ) $f3 ; done

brings a very long list of my indices, most of them being over version 7. Only system indices are missing

7

Note .async-seach has zero docs. Thats annoying. I presume (I forget) it wont let you just delete it ?

Can you paste output for the 7.x indices please.

8 
7140199 1630935915281 2021-09-06T13:45:15+00:00 .fleet-policies-7
7140199 1630927630138 2021-09-06T11:27:10+00:00 .transform-internal-007
7120099 1616702404776 2021-03-25T20:00:04+00:00 .transform-internal-006
7100099 1606833780260 2020-12-01T14:43:00+00:00 .transform-notifications-000002
7100099 1606833779860 2020-12-01T14:42:59+00:00 .transform-internal-005
7100099 1606833640118 2020-12-01T14:40:40+00:00 .kibana_security_session_1
7100099 1606833626961 2020-12-01T14:40:26+00:00 .kibana_task_manager_6
7080199 1596529255361 2020-08-04T08:20:55+00:00 .kibana_9
7080099 1592564694701 2020-06-19T11:04:54+00:00 .kibana_8
7080099 1592564693353 2020-06-19T11:04:53+00:00 .kibana_task_manager_5
7070099 1589908315713 2020-05-19T17:11:55+00:00 .async-search
7070099 1589902296397 2020-05-19T15:31:36+00:00 .apm-custom-link
7060299 1585847557125 2020-04-02T17:12:37+00:00 .kibana_task_manager_4
7050099 1575369532364 2019-12-03T10:38:52+00:00 .kibana_task_manager_3
7040199 1572013616437 2019-10-25T14:26:56+00:00 .security-7
7040199 1572008343927 2019-10-25T12:59:03+00:00 .tasks
7040199 1572008342586 2019-10-25T12:59:02+00:00 .kibana_task_manager_1
7040199 1572008342067 2019-10-25T12:59:02+00:00 .kibana_task_manager_2
7040199 1572008334117 2019-10-25T12:58:54+00:00 .apm-agent-configuration
9

And it wont let me delete it

action [indices:admin/delete] is unauthorized for user [] with effective roles [superuser,test] on restricted indices [.async-search], this action is granted by the index privileges [delete_index,manage,all]

10

Yep, weird and annoying. I'm out of good ideas. I notice:

        "routing": {
          "allocation": {
            "require": {
              "box_type": "hot"
            }
          }
        },

I don't have hot/warm/.. so don't know if that is expected.

What about a GET on

/_migration/system_features

?

11

Oh maybe this is it

{
  "features": [
    {
      "feature_name": "async_search",
      "minimum_index_version": "7.7.0",
      "migration_status": "ERROR",
      "indices": [
        {
          "index": ".async-search",
          "version": "7.7.0",
          "failure_cause": {
            "error": {
              "root_cause": [
                {
                  "type": "illegal_state_exception",
                  "reason": """unable to create new index [.async-search-reindexed-for-9] because it would match legacy templates [[{"all":{"order":0,"version":1,"index_patterns":["*"],"settings":{"index":{"number_of_replicas":"0","routing":{"allocation":{"require":{"box_type":"hot"}}}}},"mappings":{"_doc":{"_routing":{"required":false},"numeric_detection":false,"dynamic_date_formats":["strict_date_optional_time","yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"],"_source":{"excludes":[],"includes":[],"enabled":true},"dynamic":true,"date_detection":true}},"aliases":{}}}]]""",
                  "stack_trace": """org.elasticsearch.ElasticsearchException$1: unable to create new index [.async-search-reindexed-for-9] because it would match legacy templates [[{"all":{"order":0,"version":1,"index_patterns":["*"],"settings":{"index":{"number_of_replicas":"0","routing":{"allocation":{"require":{"box_type":"hot"}}}}},"mappings":{"_doc":{"_routing":{"required":false},"numeric_detection":false,"dynamic_date_formats":["strict_date_optional_time","yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"],"_source":{"excludes":[],"includes":[],"enabled":true},"dynamic":true,"date_detection":true}},"aliases":{}}}]]
	at org.elasticsearch.server@8.18.2/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:706)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:634)
	at org.elasticsearch.system_indices.action.GetFeatureUpgradeStatusResponse$IndexInfo.toXContent(GetFeatureUpgradeStatusResponse.java:306)
	at org.elasticsearch.xcontent@8.18.2/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993)
	at org.elasticsearch.xcontent@8.18.2/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:982)
	at org.elasticsearch.system_indices.action.GetFeatureUpgradeStatusResponse$FeatureUpgradeStatus.toXContent(GetFeatureUpgradeStatusResponse.java:191)
	at org.elasticsearch.xcontent@8.18.2/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993)
	at org.elasticsearch.xcontent@8.18.2/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:982)
	at org.elasticsearch.system_indices.action.GetFeatureUpgradeStatusResponse.toXContent(GetFeatureUpgradeStatusResponse.java:59)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:53)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.rest.action.RestToXContentListener.buildResponse(RestToXContentListener.java:25)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.rest.action.RestBuilderListener.buildResponse(RestBuilderListener.java:28)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.rest.action.RestResponseListener.processResponse(RestResponseListener.java:27)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.rest.action.RestActionListener.onResponse(RestActionListener.java:37)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1500)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:434)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:383)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundHandler.executeResponseHandler(InboundHandler.java:150)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:125)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:98)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:816)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:125)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:97)
	at org.elasticsearch.server@8.18.2/org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:62)
	at org.elasticsearch.transport.netty4@8.18.2/org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:55)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.codec@4.1.118.Final/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:107)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.handler@4.1.118.Final/io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1515)
	at io.netty.handler@4.1.118.Final/io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1378)
	at io.netty.handler@4.1.118.Final/io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1427)
	at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.codec@4.1.118.Final/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.transport@4.1.118.Final/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.transport@4.1.118.Final/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.transport@4.1.118.Final/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
	at io.netty.transport@4.1.118.Final/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:796)
	at io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:697)
	at io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:660)
	at io.netty.transport@4.1.118.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.common@4.1.118.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.common@4.1.118.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.lang.IllegalStateException: unable to create new index [.async-search-reindexed-for-9] because it would match legacy templates [[{"all":{"order":0,"version":1,"index_patterns":["*"],"settings":{"index":{"number_of_replicas":"0","routing":{"allocation":{"require":{"box_type":"hot"}}}}},"mappings":{"_doc":{"_routing":{"required":false},"numeric_detection":false,"dynamic_date_formats":["strict_date_optional_time","yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"],"_source":{"excludes":[],"includes":[],"enabled":true},"dynamic":true,"date_detection":true}},"aliases":{}}}]]
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:405)
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateResource(SystemIndexMigrator.java:282)
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$startFeatureMigration$7(SystemIndexMigrator.java:322)
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$updateTaskState$10(SystemIndexMigrator.java:341)
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:257)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:184)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:276)
	at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.lambda$execute$0(MasterService.java:576)
	at org.elasticsearch.cluster.service.MasterService$ExecutionResult.onPublishSuccess(MasterService.java:961)
	at org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:387)
	at org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:382)
	at org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:298)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:40)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.lang.Thread.run(Thread.java:1447)
"""
                }
              ],
              "type": "illegal_state_exception",
              "reason": """unable to create new index [.async-search-reindexed-for-9] because it would match legacy templates [[{"all":{"order":0,"version":1,"index_patterns":["*"],"settings":{"index":{"number_of_replicas":"0","routing":{"allocation":{"require":{"box_type":"hot"}}}}},"mappings":{"_doc":{"_routing":{"required":false},"numeric_detection":false,"dynamic_date_formats":["strict_date_optional_time","yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"],"_source":{"excludes":[],"includes":[],"enabled":true},"dynamic":true,"date_detection":true}},"aliases":{}}}]]""",
              "stack_trace": """java.lang.IllegalStateException: unable to create new index [.async-search-reindexed-for-9] because it would match legacy templates [[{"all":{"order":0,"version":1,"index_patterns":["*"],"settings":{"index":{"number_of_replicas":"0","routing":{"allocation":{"require":{"box_type":"hot"}}}}},"mappings":{"_doc":{"_routing":{"required":false},"numeric_detection":false,"dynamic_date_formats":["strict_date_optional_time","yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"],"_source":{"excludes":[],"includes":[],"enabled":true},"dynamic":true,"date_detection":true}},"aliases":{}}}]]
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateSingleIndex(SystemIndexMigrator.java:405)
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.migrateResource(SystemIndexMigrator.java:282)
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$startFeatureMigration$7(SystemIndexMigrator.java:322)
	at org.elasticsearch.system_indices.task.SystemIndexMigrator.lambda$updateTaskState$10(SystemIndexMigrator.java:341)
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:257)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)
	at org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:336)
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingResponseActionListener.onResponse(ActionListenerImplementations.java:184)
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:97)
	at org.elasticsearch.persistent.PersistentTasksClusterService$4.clusterStateProcessed(PersistentTasksClusterService.java:276)
	at org.elasticsearch.cluster.service.MasterService$UnbatchedExecutor.lambda$execute$0(MasterService.java:576)
	at org.elasticsearch.cluster.service.MasterService$ExecutionResult.onPublishSuccess(MasterService.java:961)
	at org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:387)
	at org.elasticsearch.cluster.service.MasterService$4.onResponse(MasterService.java:382)
	at org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:298)
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
	at org.elasticsearch.action.support.ThreadedActionListener$1.doRun(ThreadedActionListener.java:40)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.lang.Thread.run(Thread.java:1447)
12

btw, in passing, you have a mixed salad of index versions:

5x 7040199 aka 7.4.1
1x 7050099 aka 7.5.0
1x 7060299 aka 7.6.2
2x 7070099 aka 7.7.0
2x 7080099 aka 7.8.0
1x 7080199 aka 7.8.1
4x 7100099 aka 7.10.0
1x 7120099 aka 7.12.0
2x 7140199 aka 7.14.1

shows you are upgrading regularly I guess, which is commendable.

I'm sure I saw an update on this thread? Maybe you deleted it, but if you have a too-wide-applicable index template that might be a problem.

13

Yes, the conversation was marked as spam, but it is working now. Thank you so much. The new index .async-search-reindexed-for-9 matches an existing legacy index template that applies to all indices (index_patterns: ["*"]). This legacy template conflicts with the creation of the new index, preventing it from being created. I fixed it by deleting it.

14

Cool. If that legacy template was applying to all indices, double check if deleting it doesn't cost you anything, though I presume those legacy template has in meantime being over-taken by the new-style templates for almost all cases. But best to double check.

15

how did you use the API to update the parameter "verified_read_only" to "true"?. I am getting the excpetion when i try to use the below request.

curl -X PUT "/index/_settings" -H "Content-Type: application/json" -d '{
"index.verified_read_only": true
}'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"can not update private setting [index.verified_read_only]; this setting is managed by Elasticsearch"}],"type":"illegal_argument_exception","reason":"can not update private setting [index.verified_read_only]; this setting is managed by Elasticsearch"},"status":400}

16

I believe that adding a write block will do it.

17

Thanks keith for the reply. I tried that, index.blocks.write to true, but that seems to be not enough for upgrade assitant to mark this as safer to upgrade. They need this index.verified_read_only to also be set as true.

When i mark it as read only via upgrade assitant console, i have noticed this two params are set to true.

18

I had thought if you went through the block API that it would set it as verified_read_only for you but I must have been misremembering. Using the upgrade assistant is definitely the best way to go.