Upgrade from non-XPack to XPack


(Todd Lyons) #1

I have a production ELK stack with 3 master nodes, 3 data nodes, running 6.1.4. When I built the (AWS) ami's (using packer) I did not install XPack. The kibana and logstash pieces are running as docker containers in Kube. I had to rebuild the official images and uninstall XPack (this was before I knew about the *-oss images that are now available).

We have purchased an XPack license and I need to install it on our running production system (but I can't install it on our dev system to experiment since we only purchased enough seats to add to our prod ELK stack). It's not clear what steps I need to do this in order to achieve no downtime. https://www.elastic.co/guide/en/x-pack/6.1/installing-xpack.html does not tell me if my non-XPack Kibana and Logstash will or will stop working once I have my ES cluster running with XPack. It describes creating a new ELK stack, not upgrading existing non-XPack stack.

My expectation is that I need to:

  1. Create new 6.1.4 ami with xpack installed (and license pre-configured?)
  2. Do a rolling upgrade from 6.1.4 w/o xpack to 6.1.4 with xpack on my masters.
  3. Do a rolling upgrade from 6.1.4 w/o xpack to 6.1.4 with xpack on my data nodes.
  4. Use stock 6.1.4 Kibana docker image. (somehow configure the license)
  5. I don't want to configure nor use XPack on any of the logstash pods; we only purchased XPack for use in Kibana (AD Authentication).

Can anybody confirm or comment on this series of steps before I blow up my running production cluster? Am I missing anything?

What I have done so far:
First I figured I would try to get the stock Kibana 6.1.4 docker image running against my current non-XPack ES 6.1.4 cluster. I tried to run stock Kibana 6.1.4 leaving XPack installed but completely disabled. I have concluded that it cannot be done; the only way to completely disable it is to uninstall it. Otherwise, some part of the plugin detects that my 6.1.4 ES nodes don't have XPack installed, and stops me at the Status Breakdown screen. I have tried adding to the commandline:

--xpack.graph.enabled=false --xpack.ml.enabled=false --xpack.reporting.enabled=false 
--xpack.security.enabled=false --xpack.watcher.enabled=false --xpack.logstash.enabled=false 
--xpack.searchprofiler.enabled=false --xpack.tilemap.enabled=false

But when the Kibana container finishes optimizing and fully starts (about 6 minutes on my local machine, haven't tested in prod Kube clusters yet), it won't let me click on anything, it just displays the Status Breakdown screen that shows which module(s) had a problem. The above commandline options do successfully disable all of the xpack modules except for one: plugin:xpack_main@6.1.4. If I try to disable that one, Kibana fails to start because the optimization process blows up with a 500 ISE trying to import main_xpack pieces.

Other option?
To REALLY complicate this, I see that ELK 6.3+ comes with a built in "basic" license. Would I be better to update my entire stack to 6.3 or 6.4 first? At that point, XPack basic would be automatically configured for me (I think). Or does that open a different can of worms?

Remember, my primary goal here is near zero downtime. I know my RabbitMQ can handle 10 million messages getting piled up in the queue, but i haven't gone beyond that. So I figure I can handle an hour of outage without too much problem.

I'm trying to figure out what is best?

  • Rolling Upgrade ES from 6.1.4 to 6.3.x? (assume using basic license). Upgrade kibana and logstash to appropriate *-oss images.
  • Rolling Upgrade from 6.1.4 to 6.4.x? (assume using basic license). Upgrade kibana and logstash to appropriate *-oss images.
  • Upgrade 6.3|6.4 to use the purchased XPack full license. Upgrade kibana docker image to stock xpack version (and configure purchased license?)

Side question:
Does purchasing an XPack license come with support or is it expected to come from this forum? (Or if is a +1 purchase, I don't know if we paid for that.)

(Christian Dahlqvist) #2

The commercial features come as part of a subscription, which does include support, so you should be able to reach out to support for this.

(Todd Lyons) #3

Thanks Christian, I have begun that registration process.

Small change: we're going to upgrade our ELK clusters from 6.1.4 to 6.4.3 before working to integrate XPack. I know that 6.5.0 got released yesterday or today, but we'll let it curate a little bit and see what issues others have before upgrading our ELK stacks.

(Tim Vernum) #4

You should be entitled to a dev/test license to accompany your prod license. You should discuss this with your support and/or sales representative.

(Todd Lyons) #5

Thanks for the answer Tim. I finally created my account with Elastic and do indeed I see two licenses: a non-production and a production license.

Last week we upgraded our dev ELK cluster (3 nodes, combined master + data) to 6.4.3 (ES, kibana, and logstash). For ES, we installed the debs on an Ubuntu LTS ami and snapshotted it to make deployable ami's (using chef-solo via packer). For kibana, we used the official Elastic kibana docker image with no changes. For logstash, we started with the official Elastic logstash docker image, then added some additional plugins and made a self-hosted docker image in our Artifactory docker repo. We stumbled through a TLS problem and ultimately had to also control which version of the logstash-input-tcp plugin was installed (5.1.0) because the most recent version (5.2.0) doesn't like something about our custom certs (both custom CA and cert).

This morning we completed the full upgrade of our prod ELK stack (3 masters + 3 data nodes, kibana containers, and several logstash containers) to 6.4.3. It was about 3 hrs 45 mins from beginning to end, and we spent a half hour dealing with an AWS issue (outage?) trying to get IAM keys for terraform.

This officially concludes our upgrade from a non-XPack stack to an XPack stack. Now to move on and configure the actual license. We'll contact our Elastic support engineer and more forward with that.

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.