Can't start kibana 5.6.3

[mathias]

Hi,

I have been running Kibana 5.2.2 for a while without problem.
Yesterday I decided to upgrade my ELK stack to 5.6.3.
I upgraded Elasticsearch and logstash first.
They worked will with the older Kibana.
But when I tried to upgrade Kibana I says "Status:Red"

At the same time I see the following in elasticsearch log:

[2017-10-27T07:06:01,040][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [VtQPXP_] failed to put mappings on indices [[[.kibana/hQDXSt9ySLe7uEtLc3_NGA]]], type [timelion-sheet]
java.lang.IllegalArgumentException: mapper [hits] cannot be changed from type [long] to [integer]
at org.elasticsearch.index.mapper.MappedFieldType.checkTypeName(MappedFieldType.java:151) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.index.mapper.MappedFieldType.checkCompatibility(MappedFieldType.java:163) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:128) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.index.mapper.FieldTypeLookup.copyAndAddAll(FieldTypeLookup.java:94) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:400) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.index.mapper.MapperService.internalMerge(MapperService.java:336) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:268) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:311) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.service.ClusterService.executeTasks(ClusterService.java:634) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.service.ClusterService.calculateTaskOutputs(ClusterService.java:612) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.service.ClusterService.runTasks(ClusterService.java:571) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.service.ClusterService$ClusterServiceTaskBatcher.run(ClusterService.java:263) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:247) [elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:210) [elasticsearch-5.6.3.jar:5.6.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]

Deleting my .kibana index solves the problem but then all my dashboards are gone.
After that I could restart my kibana docker and the dashboards are loaded again without problem.

Is the problem related to converting old index to new format?

Kind Regards
Mathias

[Joe_Fleming]

It looks like you're running into this problem: Upgrade issue with Elastic Stack 5.6.0, workaround option until fix is available (there's a fix available there).

Additionally, you could migrate your existing Kibana index to 6.0, and it would work correctly in 5.6. See https://www.elastic.co/guide/en/kibana/6.0/migrating-6.0-index.html.

The problem is that, historically, Kibana wouldn't create any of the saved object mappings until the first time a given object was created. If something created a saved object outside of Kibana (like Beats, for example), the mappings it would create would be a little off. In 5.6 (or possibly 5.5, I forget when we added this...), Kibana started creating mappings on startup. The message you are seeing is because the mapping it's trying to set already exists, and isn't the same mapping that Kibana is trying to set.

[mathias]

@Joe_Fleming

na wouldn't create any of the saved object mappings until the first time a given object was created. If something created a saved object outside of Kibana (like Beats, for example), the mappings it would create would be a little off. In 5.6 (or possibly 5.5, I forget when we added this...), Kibana started creating mappings on startup. The message you are seeing is because the mapping it's trying to set already exists, and isn't the same mapping that Kibana is trying to set.

Thanks Joe,

I found a way around the problem.
I wait until kibana is started using your non documented rest API.
Then Set default index and timestamp.
After this I load beats templates and dashboards.
A bit inconvenient though and I am not sure it will be future proof.

Br Mathias

[Joe_Fleming]

@mathias

I wait until kibana is started using your non documented rest API.

Then Set default index and timestamp.

That's more or less the workaround we've been recommending for a while. We tell users to go create all the different kind of saved objects so that Kibana can create the mappings, but as long as you got the mapping right yourself, that's fine.

Going forward, Kibana should take care of all the mappings (and any required transformations, transparently or possibly through an upgrade tool), so if things are running correctly now, you should be ok going forward.