# Jackson error on startup of upgraded 5.6 cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 28, 2018, 2:00am UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714 "2018-06-28T02:00:53Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![eswenson](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eswenson](https://discuss.elastic.co/u/eswenson)\ **Post date:** [June 28, 2018, 2:00am UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714/1 "2018-06-28T02:00:53Z")
I have a 2.4 cluster that I'm trying to upgrade to 5.6. I've followed the documentation (including making sure clustermigration helper plugin says things are ok) and upon startup of the 5.6 cluster, I get a jackson XML parsing error associated with an apparent \_cluster/settings request. I'm not making any requests at all, so this is something that is happening as part of ES startup. As a result of the error, the master shuts down and ES is unavailable. I see this in the logs: ``` [2018-06-28 00:09:17,173][INFO][cluster.routing.allocation] [Gailyn Bailey] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logs-authx-syslog][3], [logs-authx-syslog][4], [logs-authx-syslog][0], [.kibana][0]] ...]). [2018-06-28 00:09:32,042][INFO][cluster.routing.allocation] [Gailyn Bailey] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana][0]] ...]). [2018-06-28 00:36:52,165][WARN][rest.suppressed] path: /_cluster/settings, params: {} com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input within/between Object entries at [Source: [B@2d89ddb0; line: 1, column: 25] at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:483) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon2(UTF8StreamJsonParser.java:3096) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon(UTF8StreamJsonParser.java:3008) at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:780) at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.j ava:268) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:246) at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:209) at org.elasticsearch.rest.action.admin.cluster.settings.RestClusterUpdateSettingsAction.handleRequest(RestClusterUpdateSettingsAction.java:54) ``` The exception stacktrace goes on and then a second error: ``` [2018-06-28 00:44:09,202][WARN][rest.suppressed] path: /_cluster/settings, params: {} com.fasterxml.jackson.core.JsonParseException: Unexpected character ('.' (code 46)): was expecting a colon to separate field name and value at [Source: [B@1aca8d3e; line: 1, column: 23] ``` And then finally the master shuts itself down: ``` [2018-06-28 00:45:17,344][INFO][cluster.routing.allocation.decider] [Gailyn Bailey] updating [cluster.routing.allocation.enable] from [ALL] to [NONE] [2018-06-28 00:47:14,384][INFO][node] [Gailyn Bailey] stopping ... [2018-06-28 00:47:14,527][INFO][node] [Gailyn Bailey] stopped [2018-06-28 00:47:14,527][INFO][node] [Gailyn Bailey] closing ... [2018-06-28 00:47:14,529][INFO][node] [Gailyn Bailey] closed ``` Searching on the web leads to nothing that helps -- since this error appears to come up after requests are coming into the ES cluster. In my case, no other host/process knows about this node and hence there can be no inbound traffic to the cluster. It must be a self-generated message. I updated the log settings to "debug" in log4j2.properties, but got no additional logging. Any help would be appreciated. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 28, 2018, 5:44am UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714/2 "2018-06-28T05:44:56Z")
@tlrx could you help? ---
**Author:** ![tanguy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tanguy/32/6030_2.png) [@tanguy](https://discuss.elastic.co/u/tanguy)\ **Post date:** [June 28, 2018, 7:39am UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714/3 "2018-06-28T07:39:44Z")
It seems that a Cluster Update Settings request is sent to your cluster: [2018-06-28 00:44:09,202][WARN][rest.suppressed] path: /\_cluster/settings, params: {} And this request is not correctly formatted: com.fasterxml.jackson.core.JsonParseException: Unexpected character ('.' (code 46)): was expecting a colon to separate field name and value You should try to identify which process is sending this REST request. There's something that disable shard allocation: [2018-06-28 00:45:17,344][INFO][cluster.routing.allocation.decider] [Gailyn Bailey] updating [cluster.routing.allocation.enable] from [ALL] to [NONE] And the master node should not shutdown itself as it can only be done by sending a Ctrl-C / SIGTERM event to the JVM. ---
**Author:** ![eswenson](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eswenson](https://discuss.elastic.co/u/eswenson)\ **Post date:** [June 28, 2018, 4:25pm UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714/4 "2018-06-28T16:25:27Z")
But there are no other processes running that would make requests to ES. All other ES nodes are shut down. This request must be originating from ES itself. And why would master shut down. All requests like \_cat/health hang (no response). ---
**Author:** ![eswenson](https://avatars.discourse-cdn.com/v4/letter/e/9fc348/32.png) [@eswenson](https://discuss.elastic.co/u/eswenson)\ **Post date:** [June 29, 2018, 12:50am UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714/5 "2018-06-29T00:50:31Z")
I suspect that while I did shut down all ES nodes, that I may have left up an old (circa ES 2.4) kibana running on one of the nodes. I made sure all kibana instances were down. And now I'm able to bring up the cluster. It would have been nice if ES told me from where the /\_cluster/settings request was coming from. I turned on debug level logging and there was no clue about this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 27, 2018, 12:50am UTC](https://discuss.elastic.co/t/jackson-error-on-startup-of-upgraded-5-6-cluster/137714/6 "2018-07-27T00:50:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.