Upgrade issue with Elastic Stack 5.6.0, workaround option until fix is available

Elastic Stack Kibana
1

Since the release of Elastic Stack 5.6.0, we’ve discovered an issue during the upgrade process to this version.

Who is affected

Anyone upgrading a cluster that has indexes created with Elasticsearch 2.x or prior. This does not affect users installing a fresh 5.6 cluster nor those who started their Elastic journey with the 5.x releases.

Symptom

The problem is most visible through Kibana. After attempting to login to Kibana post-upgrade, the following error appears:

1d38a818-20d1-40ca-863d-e75f815c978b
1d38a818-20d1-40ca-863d-e75f815c978b.png941×209 17.2 KB

It may also manifests as the following error:

8dcb4f04-8dc2-4b8d-92d7-d823764595b3
8dcb4f04-8dc2-4b8d-92d7-d823764595b3.jpeg998×138 18.9 KB

Or if X-Pack is in use:

99490844-2e09-4ec1-9004-1c536ee46d86
99490844-2e09-4ec1-9004-1c536ee46d86.png911×462 51.4 KB

In any case, the Elasticsearch cluster itself (including all of your data in Elasticsearch) is not affected and correctly reports itself as in Green status.

Mitigation

We have created a hotfix script you can use to fix the issues in your cluster. The script must be run on a Linux system with network access to an instance in the impacted Elasticsearch cluster. Alternatively, waiting for 5.6.1 is also an option.

Using the hotfix script:

1 - Stop Kibana

Stop the existing Kibana process using the appropriate command for your system. For example on a systemd-based Linux distribution installed via our RPM/DEB packages:

sudo systemctl stop kibana

2 - Download the hotfix script

The patch is available to download here. You can download directly to your workstation or server with curl, for example:

curl -L -O https://download.elastic.co/downloads/kibana/8134-patch.sh

3 - Run the hotfix

Without X-Pack Installed

If you are not using X-Pack then you need only specify a URL to a node in your cluster, for example:

./8134-patch.sh http://mycluster:9200

If you do not specify a URL, it defaults to http://localhost:9200

With X-Pack Installed

If you are using X-Pack and have the Security enabled, you'll need to pass credentials for a cluster user (assigned to the superuser built-in role), for example:

./8134-patch.sh https://mycluster:9200 elastic:mypassword

4 - Start Kibana

Start the existing Kibana process using the appropriate command for your system. For example on a systemd-based Linux distribution installed via our RPM/DEB packages:

sudo systemctl start kibana

Cause

The root cause is a bug in Elasticsearch we are targeting to fix in 5.6.1, and that release will be imminent. Waiting for 5.6.1 is a viable option if you have not already upgraded.

1 Like
3

The Elastic Stack 5.6.1 was released today; https://www.elastic.co/downloads

4

I have this same issue but started with a 5.x system

Upgrade from 5.5.2 to 5.6.1

5

@cutch69 I just went through that case and everything worked fine for me. Here's what I did;

Installed the full 5.5.2 stack on Ubuntu VM. Includes Elasticsearch, Kibana, Logstash, and 3 of the beats, with X-Pack installed in Elasticsearch, Kibana, and Logstash (but should all work fine without X-Pack).

The I upgraded, first Elasticsearch to 5.6.1, started Kibana and made sure things were working.
Then I stopped Kibana and upgraded it to 5.6.1 and tested again.
I didn't upgrade or restart logstash or the beats since they didn't seem to be relevant to this issue.

root@packer-virtualbox-iso-1501424719:~# history
    1  service logstash stop
    2  service metricbeat stop
    3  service packetbeat stop
    4  service filebeat stop
    5  service kibana stop
    6  service elasticsearch stop
    7  /usr/share/elasticsearch/bin/elasticsearch-plugin remove xpack  (wrong plugin name)
    8  /usr/share/elasticsearch/bin/elasticsearch-plugin remove x-pack
    9  cp /etc/elasticsearch/elasticsearch.yml /tmp/
   10  wget http://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.1.deb
   11  dpkg -i elasticsearch-5.6.1.deb
   12  wget http://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.6.1.zip
   13  pwd
   14  /usr/share/elasticsearch/bin/elasticsearch-plugin install -b file:////home/vagrant/x-pack-5.6.1.zip
   15  service elasticsearch start
   16  service kibana start
(I tested here and everything was working)
   17  service kibana stop
   18  wget http://artifacts.elastic.co/downloads/kibana/kibana-5.6.1-amd64.deb
   19  /usr/share/kibana/bin/kibana-plugin remove x-pack
   20  dpkg -i kibana-5.6.1-amd64.deb
   21  /usr/share/kibana/bin/kibana-plugin install file:////home/vagrant/x-pack-5.6.1.zip
   22  service kibana start
   23  history
6

I upgraded to 5.6.0 and hit this issue, but successfully applied the hotfix, 8132-patch.sh. At this point Kibana was happy and working as expected.

I then upgraded ES and Kibana to 5.6.1. Now Kibana is back in a "Red" state with some different errors. ES cluster status is "Green". Is this related to the hotfix patch? Or something completely new?

kibanared
kibanared.png1032×497 70.9 KB

I also see this show up in Kibana's stdout log over and over.

{"type":"log","@timestamp":"2017-09-20T10:57:28Z","tags":["info","elasticsearch"],"pid":17028,"typeName":"url","typeMapping":{"properties":{"accessCount":{"type":"long"},"accessDate":{"type":"date"},"createDate":{"type":"date"},"url":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":2048}}}}},"message":"Adding mappings to kibana index for SavedObject type \"url\""}

No timestamps in the stderr log, but here's a snippet from that:

Error while executing search { Error: Request Timeout after 30000ms
    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:336:15
    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:365:7)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)
  status: undefined,
  displayName: 'RequestTimeout',
  message: 'Request Timeout after 30000ms',
  body: undefined }
Unhandled rejection Error: Request Timeout after 30000ms
    at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:336:15
    at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:365:7)
    at ontimeout (timers.js:365:14)
    at tryOnTimeout (timers.js:237:5)
    at Timer.listOnTimeout (timers.js:207:5)
Unhandled rejection Error: No Living connections
    at sendReqWithConnection (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:207:15)
    at next (/usr/share/kibana/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
    at _combinedTickCallback (internal/process/next_tick.js:67:7)
    at process._tickDomainCallback (internal/process/next_tick.js:122:9)
7

Yeah i figure out what the issue was i had to upgrade both kibana servers. seems odd

8

I have installed 5.6.1 and I'm getting the xpack issue, I've tried to apply the hot fix above with no joy.

Is the old patch version specific and there is a seperate issue with 5.6.1 that still needs to be addressed?

9

After the 5.6.1 upgrade (and before that 5.6.0 + 8134-patch.sh workaround) I'm seeing this:

[2017-09-21T09:17:51,091][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [elkelastic3] failed to put mappings on indices [[[.kibana-6/LaISONzyRfGlu2gzSLxvhA]]], type [url]
java.lang.IllegalArgumentException: [url] is defined as a field in mapping [url] but this name is already used for an object in other types

This appears to be related to the patch + 5.6.1. Anyone know the safe/recommended way to recover from this?

10

@Ant & @benpolzin ,

Would you mind separately posting issues outlining the process you went through? Including, what versions you upgrade to, if and when you ran the script, etc.

Mention me and I will work with you through it.

Thanks and apologies for any inconvenience you are experiencing.

11

@tylersmalley As I did this on a dev system I've just gone to check with the devs that Elastic is working for them even is kibana isn't and they told me "what are you on about I was using it Friday afternoon". I've done a Ctrl+Shift+R on the kibaba sign in page and I'm no longer getting the error, however in the interests of completeness (and because numpty boy had already typed it out before asking the devs if they saw any issues with the elastic service) this is the process I followed incase it can help anyone else.

I upgraded from 5.4.3 straight to 5.6.1 after doing so when I tried to sign into Kibana I was told

"Login is currently disabled because the license could not be determined. Please check that Elasticsearch has the X-Pack plugin installed and is reachable, then refresh this page."

I then re-registered the license file with no joy after this I came on the forum and saw the above post and downloaded and ran the 8134-patch.sh file but still get the same message about login being disabled.

waited 5 days and then had a dev tell me what am I talking about it's working fine...

The process I went through to do the 5.4.3 to 5.6.1 upgrade was:
disable cluter routing
remove x-pack for both elastic and kibana
stop elastic and kibana
yum install the new version for elastic and kibana
install xpack for elastic and kibana
start elastic
start kibana
re-enable cluster routing

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.