RPM and DEB packages impractical for use

[markdaku]

Using the repo version of elastic products has proven to be a no go.

For the very simple reason that if you use the method of update repo list and then upgrade packages you will be offline until you perform remediation.

For a few years now I have tried to make this work. It simply does not. Each release brings something new to the table and something else needs a little more hands on attention.

Example for deb:
Prompt> apt-get update
Prompt> apt-get upgrade
Instantly broken env. You are dropping events the second this happens. This should NEVER be the case. Lost events means lost trust in monitoring of target ENV's.

First we have the repos with out of date certs. Secondly a package update does NOT update internal configuration of DB or addons. Which results in LOST EVENTS. These are very simple problems but are consistently neglected with every single release.

I would like to propose a program of work that actually does make this work. I do love Elastic, But this upgrade patch path is unbelievably fragile. This needs to be a priority 1 type feature for upcomgin releases.

[warkolm (Mark Walkom)]

@markdaku

First we have the repos with out of date certs.

https://artifacts.elastic.co doesn't appear to be out of date?

@markdaku

Secondly a package update does NOT update internal configuration of DB or addons.

Can you elaborate on what you mean here please?

[markdaku]

@warkolm

@markdaku

First we have the repos with out of date certs.

https://artifacts.elastic.co doesn't appear to be out of date?

At the moment is it. However quite frequently the cert is downgraded and this results in errors. On many other occasions the repo fails to return a sources.list. The forums discuss these issues on a regular basis. As a result this will often break the update process in general.

@warkolm

@markdaku

Secondly a package update does NOT update internal configuration of DB or addons.

Can you elaborate on what you mean here please?

The very own Elastic Upgrade instructions actually fully illustrate the issue.
First off indices need to be upgraded.
If you use plugins the interface instantly STOPS. Instant loss of data.
If for example a beat upgrades prior to elasticsearch even a minor upgrade that event data is LOST.

Since services on various hosts may upgrade or patch in different orders do to various business concerns and since even minor updates to elastic components result in dropped event data until Operational upgrade tasks are completed.

Elastic stack does not dictate the upgrade order of most environments services. The actually business services do. Thus if one uses the DEB or RPM packages for elastic stack you will have services that suddenly loose event data. And this happens EVERY SINGLE TIME. If a patch comes out we have to instantly address the entire elast stack topology and upgrade everything and validate all components in the elastic stack.

Thus this mandates a few things. 1. We can't use the official repos directly. If we stick to deb /rpm we must clone the releases to a local repo first. 2. We can never use latest greatest stable. As the upgrade process ALWAYS drops event data. Thus we have to specifically schedule upgrade windows for the elastic stack services. And we can ONLY upgrade the elastic stack in those windows.

[markdaku]

I operate with a very simple rules.

1. Our product must be in OS specific packaging.
2. Upgrades using standard upgrade tools should NEVER break the product.

Thus commands like
apt update
apt upgrade

Should always result in a the existing software to be in the same sain state it was in prior to the upgrade. If it's a service and running prior. IT should be in a running state after.

If it is a breaking release this mandates a major version change. And the new version of product should install parallel to the previous install. It should be in a default non running state and it should warn that additional operational tasks are required post installation in order to use the new version. Leaving the old version untouched.

Thus if an upgrade occures and a plugin breaks as a result. The new plugin should not be in an run state. The old plugin should remain in place and in an operational state. etc.

[warkolm (Mark Walkom)]

@markdaku

At the moment is it

SSL Certificate Checker shows that the cert is valid until March next year. This also shows the same thing;

$ echo | openssl s_client -showcerts -servername  https://artifacts.elastic.co -connect artifacts.elastic.co:443 2>/dev/null | openssl x509 -noout -dates
notBefore=Jan 18 00:00:00 2016 GMT
notAfter=Mar 27 12:00:00 2019 GMT

If you can you show logs or other output of what you are seeing then it'll help identify where the issue is.

@markdaku

First off indices need to be upgraded.

Only for indices created on version N-2 (where N is the major version you are moving to).

@markdaku

If you use plugins the interface instantly STOPS. Instant loss of data.

You do need to upgrade plugins, yes. That overall process is not ideal and it's something we want to improve.

@markdaku

If for example a beat upgrades prior to elasticsearch even a minor upgrade that event data is LOST.

That should not be the case. Can you share more information, logs, etc for this?

[markdaku]

Also Note: Kibana fails to start after 6.3 upgrade by package. Upgrade instructions are insufficient. You are left with an inoperative installation.

kibana-plugin remove x-pack fails. This task must be done by hand.

Also the optimise babel cache is owned by the wrong user. This also blocks the startup.

[warkolm (Mark Walkom)]

While I appreciate your frustrations, unless you're able to provide logs it's difficult to provide further assistance.