Encountering "only value lists are allowed in serialized settings" exception after upgrade to 6.7.0 from 5.6.5

ParitoshBh · April 3, 2019, 5:41pm

I performed a rolling upgrade in a cluster of 3 nodes, moving from version 5.6.5 to 6.7.0 and ended up seeing the following exception in log file,

[2019-04-03T12:04:06,926][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:1, file:/var/lib/elasticsearch/nodes/0/_state/global-1.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.io.IOException: failed to read [id:1, file:/var/lib/elasticsearch/nodes/0/_state/global-1.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.7.0.jar:6.7.0]
        ... 15 more
Caused by: java.lang.IllegalStateException: only value lists are allowed in serialized settings
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:692) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:650) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:640) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:443) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1253) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.7.0.jar:6.7.0]
        ... 15 more

Here's the output of ls -l /var/lib/elasticsearch/nodes/0/_state/,

-rw-r--r--. 1 elasticsearch elasticsearch 67089 Apr  3 12:05 global-1.st
-rw-r--r--. 1 elasticsearch elasticsearch    71 Apr  3 12:04 node-0.st

Here's what I have in elasticsearch.yml,

cluster.name: search
node.name: node-1

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host:
  - 172.16.90.94
  - _local_

discovery.zen.ping.unicast.hosts:
  - 172.16.90.95

I wasn't using any subscription earlier, no ml jobs, etc.

If I clear out the /var/lib/elasticsearch/nodes/ path then node comes back without complaining,

sudo rm -rfv /var/lib/elasticsearch/nodes/*

Any help will be appreciated

DavidTurner · April 3, 2019, 5:50pm

Hmm, this seems like a bug. Could you share /var/lib/elasticsearch/nodes/0/_state/global-1.st so we can investigate further?

ParitoshBh · April 3, 2019, 5:51pm

How do you want me to share the file? Email?

DavidTurner · April 3, 2019, 5:51pm

Sure if that works for you; my email address is david.turner@elastic.co.

ParitoshBh · April 3, 2019, 6:21pm

Done!

DavidTurner · April 3, 2019, 7:05pm

Thanks. I haven't quite worked out the issue yet, but it seems to be to do with a template called REDACTED-catalog-template which has an analyser defined under index.analysis.analyzer.brand_replacement.char_filter that is the location of the error. Do you know how this analyser was defined in your 5.6 cluster?

ParitoshBh · April 3, 2019, 7:24pm

Here's how it was defined in 5.6 (as in-line char filter),

          'brand_replacement' => [
              'tokenizer' => 'brand_replacement',
              'filter' => ['trim', 'lowercase'],
              'char_filter' => [
                 [
                   'type' => 'pattern_replace',
                   'pattern' => '-',
                   'replacement' => ' '
                  ]
              ]
          ]

Edit 1:

I got rid of the template and restarted the service. It came up just fine. I understand defining in-line char_filter isn't supported/allowed anymore but the error message hinted at misconfigured settings.

Also, would you mind sharing how you pin-pointed the cause of error?

DavidTurner · April 3, 2019, 7:44pm

Yes, I think this might be a bug. Rolling upgrades from 5.6 to 6.x are supported, and there is some code to handle format changes like this, but it seems that they are not handling this correctly.

Sure, but I will admit it wasn't particularly elegant. I added the following test case to a random test suite:

    public void testDiscussIssue() throws IOException {
        MetaData.FORMAT.read(xContentRegistry(), Paths.get("/Users/davidturner/discuss/175254/global-1.st"));
    }

Then I ran this test in my IDE and got the same exception you reported:

java.lang.IllegalStateException: only value lists are allowed in serialized settings

	at __randomizedtesting.SeedInfo.seed([71154C66D7CF5377:C3048DA835B3628D]:0)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:692)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:650)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:640)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:443)
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1253)
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300)
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291)
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196)
	at org.elasticsearch.gateway.GatewayMetaStateTests.testDiscussIssue(GatewayMetaStateTests.java:571)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...

Then I put a breakpoint on the line that throws this exception and poked around until I'd found the path to the setting and the template name.

ParitoshBh · April 3, 2019, 7:52pm

Not sure if this is indeed a bug (and a github issue needs be created) but I'll mark this thread as resolved since the core issue was identified at-least.

Thanks again for prompt response!

DavidTurner · April 3, 2019, 9:21pm

Thanks for the report @ParitoshBh, I opened https://github.com/elastic/elasticsearch/issues/40815.

system · May 1, 2019, 9:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.