Jackson error on startup of upgraded 5.6 cluster

[eswenson (Eric Swenson)]

I have a 2.4 cluster that I'm trying to upgrade to 5.6. I've followed the documentation (including making sure clustermigration helper plugin says things are ok) and upon startup of the 5.6 cluster, I get a jackson XML parsing error associated with an apparent _cluster/settings request. I'm not making any requests at all, so this is something that is happening as part of ES startup. As a result of the error, the master shuts down and ES is unavailable.

I see this in the logs:

[2018-06-28 00:09:17,173][INFO ][cluster.routing.allocation] [Gailyn Bailey] Cluster health status     changed from [RED] to [YELLOW] (reason: [shards started [[logs-authx-syslog][3], [logs-authx-syslog][4],     [logs-authx-syslog][0], [.kibana][0]] ...]).
[2018-06-28 00:09:32,042][INFO ][cluster.routing.allocation] [Gailyn Bailey] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.kibana][0]] ...]).
[2018-06-28 00:36:52,165][WARN ][rest.suppressed          ] path: /_cluster/settings, params: {}
com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input within/between Object     entries
 at [Source: [B@2d89ddb0; line: 1, column: 25]
    at     com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:483)
    at     com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon2(UTF8StreamJsonParser.java:3096)
    at     com.fasterxml.jackson.core.json.UTF8StreamJsonParser._skipColon(UTF8StreamJsonParser.java:3008)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:780)
    at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.j    ava:268)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.readMap(AbstractXContentParser.java:246)
    at org.elasticsearch.common.xcontent.support.AbstractXContentParser.map(AbstractXContentParser.java:209)
    at     org.elasticsearch.rest.action.admin.cluster.settings.RestClusterUpdateSettingsAction.handleRequest(RestClusterUpdateSettingsAction.java:54)

The exception stacktrace goes on and then a second error:

[2018-06-28 00:44:09,202][WARN ][rest.suppressed          ] path: /_cluster/settings, params: {}
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('.' (code 46)): was expecting a    colon to separate field name and value
at [Source: [B@1aca8d3e; line: 1, column: 23]

And then finally the master shuts itself down:

[2018-06-28 00:45:17,344][INFO ][cluster.routing.allocation.decider] [Gailyn Bailey] updating     [cluster.routing.allocation.enable] from [ALL] to [NONE]
[2018-06-28 00:47:14,384][INFO ][node                     ] [Gailyn Bailey] stopping ...
[2018-06-28 00:47:14,527][INFO ][node                     ] [Gailyn Bailey] stopped
[2018-06-28 00:47:14,527][INFO ][node                     ] [Gailyn Bailey] closing ...
[2018-06-28 00:47:14,529][INFO ][node                     ] [Gailyn Bailey] closed

Searching on the web leads to nothing that helps -- since this error appears to come up after requests are coming into the ES cluster. In my case, no other host/process knows about this node and hence there can be no inbound traffic to the cluster. It must be a self-generated message.

I updated the log settings to "debug" in log4j2.properties, but got no additional logging.

Any help would be appreciated.

[dadoonet (David Pilato)]

@tlrx could you help?

[tanguy]

It seems that a Cluster Update Settings request is sent to your cluster:

[2018-06-28 00:44:09,202][WARN ][rest.suppressed ] path: /_cluster/settings, params: {}

And this request is not correctly formatted:

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('.' (code 46)): was expecting a colon to separate field name and value

You should try to identify which process is sending this REST request. There's something that disable shard allocation:

[2018-06-28 00:45:17,344][INFO ][cluster.routing.allocation.decider] [Gailyn Bailey] updating [cluster.routing.allocation.enable] from [ALL] to [NONE]

And the master node should not shutdown itself as it can only be done by sending a Ctrl-C / SIGTERM event to the JVM.

[eswenson (Eric Swenson)]

But there are no other processes running that would make requests to ES. All other ES nodes are shut down. This request must be originating from ES itself. And why would master shut down. All requests like _cat/health hang (no response).

[eswenson (Eric Swenson)]

I suspect that while I did shut down all ES nodes, that I may have left up an old (circa ES 2.4) kibana running on one of the nodes. I made sure all kibana instances were down. And now I'm able to bring up the cluster. It would have been nice if ES told me from where the /_cluster/settings request was coming from. I turned on debug level logging and there was no clue about this.