Issue while upgrading elasticsearch from 7.17.3 to 8.5.1

Elastic Stack Elasticsearch
1

I'm trying to upgrade elasticsearch from 7.17.3 to 8.5.1, but it is throwing an error: "error.type":"java.lang.IllegalArgumentException".

This is my yaml:

image
image896×370 26.7 KB

Full Error:
{"@timestamp":"2024-09-27T04:37:48.833Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-master-2","elasticsearch.cluster.name":"elasticsearch","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:151)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:56)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.node.Node.(Node.java:468)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.node.Node.(Node.java:318)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ingest] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n"}

2

That's odd. Are you sure that some node doesn't have a yaml that has that node.data setting in it? The upgrade assistant didn't report any problems with it?

3

node.data and node.master aren't settings any more. I have seen these docs for more information.
So, I have added node.roles: [master, data].
It could be great, if you looked into it and provide some solution, as this is effecting our production servers.

4

Any information on this?

5

You pointed out that you made a change to fix the error and have not posted any further errors, so I am not sure what you need information about. Did the change you made fix the issue? If not, what errors are you seeing now?

6

I'm sorry for the miscommunication. The change I made did not fix it.
I'm still seeing the same error.
{"@timestamp":"2024-09-27T04:37:48.833Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-master-2","elasticsearch.cluster.name":"elasticsearch","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:151)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:56)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.node.Node.(Node.java:468)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.node.Node.(Node.java:318)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ingest] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n"}

7

Please show you config. It looks like you still have the incorrect setting in there.

8

If you are asking for elasticsearch config, it is already pasted above. If you are asking for any other config, please describe.

9

You should be fine with the default role settings so I would recommend that you try removing node.roles completely. Also note that it is recommended to set the heap to 50% of available RAM. The latest versions of Elsticsearch do this by default, so I would also recommend not explicitly setting the heap size.

10

This is my new config

elasticsearch:
  image: "docker.elastic.co/elasticsearch/elasticsearch"
  imageTag: "8.5.1"
  imagePullPolicy: "IfNotPresent"
# Recommended heap size for elastic search is 50% of the available size
  esJavaOpts: "-Xmx128m -Xms128m"
  resources:
    requests:
      cpu: "200m"
      memory: "2Gi"
    limits:
      memory: "2Gi"

Even after making the changes you suggested, it still throwing the same error.
{"@timestamp":"2024-10-11T04:53:02.091Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch-master-2","elasticsearch.cluster.name":"elasticsearch","error.type":"java.lang.IllegalArgumentException","error.message":"unknown setting [node.ml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings","error.stack_trace":"java.lang.IllegalArgumentException: unknown setting [node.ml] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:561)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:507)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:477)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:447)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:151)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.common.settings.SettingsModule.(SettingsModule.java:56)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.node.Node.(Node.java:468)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.node.Node.(Node.java:318)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch$2.(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:214)\n\tat org.elasticsearch.server@8.5.1/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.master] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.ingest] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.remote_cluster_client] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n\tSuppressed: java.lang.IllegalArgumentException: unknown setting [node.data] please check that any required plugins are installed, or check the breaking changes documentation for removed settings\n\t\t... 11 more\n"}

11

Somewhere you have several unsupported properties: node.data, node.remote_cluster_client, node.ingest, node.master, and node.ml. Is i possible that you have a node that is connecting to the cluster that you're not aware of (although it looks like this is coming from elasticsearch-master-2 -- is that the node whose configuration you shared)?
Can you restore your 7.17.3 cluster from snapshots, and then run the upgrade assistant? If that doesn't warn you about anything, a last resort might be to run the get cluster state API to see where that setting is coming from.

1 Like
12

image
image812×648 48.7 KB

  1. The node.roles are present by default.
  2. As per the logs, it is not connecting to any container where different configurations are there. Let me know if I have to check this anywhere else.
  3. Since, it is docker elastic container, we do not have an option to upgrade from the UI.
13

Any updates on this, since there are no comments from past 6 days. @Keith_Massey

14

Hello,

It is not sure where the configuration you shared in the previous post is from as you didn't provide any context.

But for version 8 you cannot have node.data, node.ingest, node.master etc settings, you need to use node.roles: [data, ingest, master, etc] instead.

You need to change the settings you are using whatever they are.