Troubles when upgrading from 5.2.2 to 5.5.2

Elastic Stack Elasticsearch
1

Hi!

I'm adding new node to existing ES cluster. This cluster is running under 5.2.2

I'm install ES through Debian packages, so latest version for now is 5.5.2.

Node was added successfully, but when I enable relocation some shards were moved to this node, they became unassigned and I saw this error on this node:

[2017-08-25T15:02:17,690][WARN ][o.e.i.c.IndicesClusterStateService] [sample] [[smaple_index][6]] marking and sending shard failed due to [failed to create index]
java.lang.StringIndexOutOfBoundsException: String index out of range: -3
at java.lang.String.substring(String.java:1967) ~[?:1.8.0_144]
at org.elasticsearch.common.logging.DeprecationLogger.extractWarningValueFromWarningHeader(DeprecationLogger.java:247) ~[elasticsearch-5.5.2.jar:5.5.2]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[?:1.8.0_144]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[?:1.8.0_144]
at org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.putResponse(ThreadContext.java:446) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ThreadContextStruct.access$1100(ThreadContext.java:352) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext.addResponseHeader(ThreadContext.java:280) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.logging.DeprecationLogger.deprecated(DeprecationLogger.java:285) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.logging.DeprecationLogger.deprecated(DeprecationLogger.java:124) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.settings.Settings.getAsBoolean(Settings.java:348) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.index.analysis.PatternCaptureGroupTokenFilterFactory.(PatternCaptureGroupTokenFilterFactory.java:48) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.indices.analysis.AnalysisModule$1.get(AnalysisModule.java:353) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.index.analysis.AnalysisRegistry.buildMapping(AnalysisRegistry.java:342) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.index.analysis.AnalysisRegistry.buildTokenFilterFactories(AnalysisRegistry.java:171) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.index.analysis.AnalysisRegistry.build(AnalysisRegistry.java:155) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.index.IndexService.(IndexService.java:145) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.index.IndexModule.newIndexService(IndexModule.java:363) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.indices.IndicesService.createIndexService(IndicesService.java:449) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:414) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:148) ~[elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.createIndices(IndicesClusterStateService.java:444) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:202) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cluster.service.ClusterService.callClusterStateAppliers(ClusterService.java:814) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cluster.service.ClusterService.publishAndApplyChanges(ClusterService.java:768) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:587) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.5.2.jar:5.5.2]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.5.2.jar:5.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

I've downgraded this node to 5.2.2 and shards became active.

Then I've upgraded to 5.5.2 one existing node from my cluster and saw the same error.

I did not see any breaking change in index structure. What is going on and how I can fix this ?

2

Are you following the steps for a cluster upgrade?
https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
https://www.elastic.co/guide/en/elasticsearch/reference/current/rolling-upgrades.html

3

Yes, I do.

4

How many indexes do you have? Do they all fail in the new node?

5

Hi @aeryaguzov,

that's tricky. I tried now reproducing your problem locally but without any "luck". :frowning:

There seems to be a problem when formatting a deprecation log message. The deprecation log message is most likely caused by a pattern_capture token filter that uses the setting preserve_original but does not use one of the values true, false, "true", "false" (Elasticsearch also still accepts 0, 1, "off", "on", "no" and "yes" until 6.0 but that is deprecated, hence the warning).

One option to get rid of the error is to fix the configuration prior to upgrading. As a consequence, no deprecation log message will get written and you should not get caught by the bug. I'd recommend that you check your index settings and ensure that only valid representations of booleans (i.e. true, false, "true", "false") are used. See the docs how to update analysis settings on an index.

It would be great if you could share the affected configuration because this could be helpful in resolving this problem in general.

Daniel

6

Hi @danielmitterdorfer ,

Thanks for your help!

You are right. The problem was in preserve_original value which was "1".

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.