# Encountering "only value lists are allowed in serialized settings" exception after upgrade to 6.7.0 from 5.6.5

**URL:** <https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254>\
**Category:** Elasticsearch\
**Created:** [April 3, 2019, 5:41pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254 "2019-04-03T17:41:31Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [April 3, 2019, 5:41pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/1 "2019-04-03T17:41:32Z")

</div>

I performed a rolling upgrade in a cluster of 3 nodes, moving from version 5.6.5 to 6.7.0 and ended up seeing the following exception in log file,

```auto
[2019-04-03T12:04:06,926][ERROR][o.e.b.Bootstrap] [node-1] Exception
org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:1, file:/var/lib/elasticsearch/nodes/0/_state/global-1.st]
        at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:164) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:304) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.GatewayMetaState.<init>(GatewayMetaState.java:88) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.node.Node.<init>(Node.java:499) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.7.0.jar:6.7.0]
        at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.7.0.jar:6.7.0]
Caused by: java.io.IOException: failed to read [id:1, file:/var/lib/elasticsearch/nodes/0/_state/global-1.st]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:298) ~[elasticsearch-6.7.0.jar:6.7.0]
        ... 15 more
Caused by: java.lang.IllegalStateException: only value lists are allowed in serialized settings
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:692) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:650) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:640) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:443) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1253) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196) ~[elasticsearch-6.7.0.jar:6.7.0]
        at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:294) ~[elasticsearch-6.7.0.jar:6.7.0]
        ... 15 more

```

Here's the output of `ls -l /var/lib/elasticsearch/nodes/0/_state/`,

```auto
-rw-r--r--. 1 elasticsearch elasticsearch 67089 Apr 3 12:05 global-1.st
-rw-r--r--. 1 elasticsearch elasticsearch 71 Apr 3 12:04 node-0.st

```

Here's what I have in `elasticsearch.yml`,

```auto
cluster.name: search
node.name: node-1

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host:
  - 172.16.90.94
  - _local_

discovery.zen.ping.unicast.hosts:
  - 172.16.90.95

```

I wasn't using any subscription earlier, no ml jobs, etc.

If I clear out the `/var/lib/elasticsearch/nodes/` path then node comes back without complaining,

```auto
sudo rm -rfv /var/lib/elasticsearch/nodes/*

```

Any help will be appreciated 🙂

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 3, 2019, 5:50pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/2 "2019-04-03T17:50:00Z")

</div>

Hmm, this seems like a bug. Could you share `/var/lib/elasticsearch/nodes/0/_state/global-1.st` so we can investigate further?

---

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [April 3, 2019, 5:51pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/3 "2019-04-03T17:51:16Z")

</div>

How do you want me to share the file? Email?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 3, 2019, 5:51pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/4 "2019-04-03T17:51:43Z")

</div>

Sure if that works for you; my email address is [david.turner@elastic.co](mailto:david.turner@elastic.co).

---

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [April 3, 2019, 6:21pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/5 "2019-04-03T18:21:04Z")

</div>

Done!

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 3, 2019, 7:05pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/6 "2019-04-03T19:05:09Z")

</div>

Thanks. I haven't quite worked out the issue yet, but it seems to be to do with a template called `REDACTED-catalog-template` which has an analyser defined under `index.analysis.analyzer.brand_replacement.char_filter` that is the location of the error. Do you know how this analyser was defined in your 5.6 cluster?

---

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [April 3, 2019, 7:24pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/7 "2019-04-03T19:24:17Z")

</div>

Here's how it was defined in 5.6 (as in-line char filter),

```php
          'brand_replacement' => [
              'tokenizer' => 'brand_replacement',
              'filter' => ['trim', 'lowercase'],
              'char_filter' => [
                 [
                   'type' => 'pattern_replace',
                   'pattern' => '-',
                   'replacement' => ' '
                  ]
              ]
          ]

```

_Edit 1:_

I got rid of the template and restarted the service. It came up just fine. I understand defining in-line char\_filter isn't supported/allowed anymore but the error message hinted at misconfigured settings.

Also, would you mind sharing how you pin-pointed the cause of error?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 3, 2019, 7:44pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/8 "2019-04-03T19:44:23Z")

</div>

> [@ParitoshBh](#):
>
> I got rid of the template and restarted the service. It came up just fine. I understand defining in-line char\_filter isn't supported/allowed anymore but the error message hinted at misconfigured settings.

Yes, I think this might be a bug. Rolling upgrades from 5.6 to 6.x are supported, and there is some code to handle format changes like this, but it seems that they are not handling this correctly.

> [@ParitoshBh](#):
>
> Also, would you mind sharing how you pin-pointed the cause of error?

Sure, but I will admit it wasn't particularly elegant. I added the following test case to a random test suite:

```plaintext
    public void testDiscussIssue() throws IOException {
        MetaData.FORMAT.read(xContentRegistry(), Paths.get("/Users/davidturner/discuss/175254/global-1.st"));
    }

```

Then I ran this test in my IDE and got the same exception you reported:

```plaintext
java.lang.IllegalStateException: only value lists are allowed in serialized settings

	at __randomizedtesting.SeedInfo.seed([71154C66D7CF5377:C3048DA835B3628D]:0)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:692)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:681)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:650)
	at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:640)
	at org.elasticsearch.cluster.metadata.IndexTemplateMetaData$Builder.fromXContent(IndexTemplateMetaData.java:443)
	at org.elasticsearch.cluster.metadata.MetaData$Builder.fromXContent(MetaData.java:1253)
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1300)
	at org.elasticsearch.cluster.metadata.MetaData$1.fromXContent(MetaData.java:1291)
	at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:196)
	at org.elasticsearch.gateway.GatewayMetaStateTests.testDiscussIssue(GatewayMetaStateTests.java:571)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...

```

Then I put a breakpoint on the line that throws this exception and poked around until I'd found the path to the setting and the template name.

---

<div class="post-metadata">

**Author:** ![ParitoshBh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paritoshbh/32/22487_2.png) [@ParitoshBh](https://discuss.elastic.co/u/ParitoshBh)\
**Post date:** [April 3, 2019, 7:52pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/9 "2019-04-03T19:52:09Z")

</div>

Not sure if this is indeed a bug (and a github issue needs be created) but I'll mark this thread as resolved since the core issue was identified at-least.

Thanks again for prompt response!

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 3, 2019, 9:21pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/10 "2019-04-03T21:21:04Z")

</div>

Thanks for the report @ParitoshBh, I opened [https://github.com/elastic/elasticsearch/issues/40815](https://github.com/elastic/elasticsearch/issues/40815).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2019, 9:21pm UTC](https://discuss.elastic.co/t/encountering-only-value-lists-are-allowed-in-serialized-settings-exception-after-upgrade-to-6-7-0-from-5-6-5/175254/11 "2019-05-01T21:21:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.