I have a production ELK stack with 3 master nodes, 3 data nodes, running 6.1.4. When I built the (AWS) ami's (using packer) I did not install XPack. The kibana and logstash pieces are running as docker containers in Kube. I had to rebuild the official images and uninstall XPack (this was before I knew about the *-oss images that are now available).
We have purchased an XPack license and I need to install it on our running production system (but I can't install it on our dev system to experiment since we only purchased enough seats to add to our prod ELK stack). It's not clear what steps I need to do this in order to achieve no downtime. https://www.elastic.co/guide/en/x-pack/6.1/installing-xpack.html does not tell me if my non-XPack Kibana and Logstash will or will stop working once I have my ES cluster running with XPack. It describes creating a new ELK stack, not upgrading existing non-XPack stack.
My expectation is that I need to:
- Create new 6.1.4 ami with xpack installed (and license pre-configured?)
- Do a rolling upgrade from 6.1.4 w/o xpack to 6.1.4 with xpack on my masters.
- Do a rolling upgrade from 6.1.4 w/o xpack to 6.1.4 with xpack on my data nodes.
- Use stock 6.1.4 Kibana docker image. (somehow configure the license)
- I don't want to configure nor use XPack on any of the logstash pods; we only purchased XPack for use in Kibana (AD Authentication).
Can anybody confirm or comment on this series of steps before I blow up my running production cluster? Am I missing anything?
What I have done so far:
First I figured I would try to get the stock Kibana 6.1.4 docker image running against my current non-XPack ES 6.1.4 cluster. I tried to run stock Kibana 6.1.4 leaving XPack installed but completely disabled. I have concluded that it cannot be done; the only way to completely disable it is to uninstall it. Otherwise, some part of the plugin detects that my 6.1.4 ES nodes don't have XPack installed, and stops me at the Status Breakdown screen. I have tried adding to the commandline:
--xpack.graph.enabled=false --xpack.ml.enabled=false --xpack.reporting.enabled=false
--xpack.security.enabled=false --xpack.watcher.enabled=false --xpack.logstash.enabled=false
--xpack.searchprofiler.enabled=false --xpack.tilemap.enabled=false
But when the Kibana container finishes optimizing and fully starts (about 6 minutes on my local machine, haven't tested in prod Kube clusters yet), it won't let me click on anything, it just displays the Status Breakdown screen that shows which module(s) had a problem. The above commandline options do successfully disable all of the xpack modules except for one: plugin:xpack_main@6.1.4. If I try to disable that one, Kibana fails to start because the optimization process blows up with a 500 ISE trying to import main_xpack pieces.
Other option?
To REALLY complicate this, I see that ELK 6.3+ comes with a built in "basic" license. Would I be better to update my entire stack to 6.3 or 6.4 first? At that point, XPack basic would be automatically configured for me (I think). Or does that open a different can of worms?
Remember, my primary goal here is near zero downtime. I know my RabbitMQ can handle 10 million messages getting piled up in the queue, but i haven't gone beyond that. So I figure I can handle an hour of outage without too much problem.
I'm trying to figure out what is best?
- Rolling Upgrade ES from 6.1.4 to 6.3.x? (assume using basic license). Upgrade kibana and logstash to appropriate *-oss images.
- Rolling Upgrade from 6.1.4 to 6.4.x? (assume using basic license). Upgrade kibana and logstash to appropriate *-oss images.
- Upgrade 6.3|6.4 to use the purchased XPack full license. Upgrade kibana docker image to stock xpack version (and configure purchased license?)
Side question:
Does purchasing an XPack license come with support or is it expected to come from this forum? (Or if is a +1 purchase, I don't know if we paid for that.)