Build Quality/Lack of testing

Elastic Stack Kibana
1

Hi

I have just upgraded ElasticSearch 5.5.2 on a simple 2 node Windows cluster. Running Kibana in discover works only for a few seconds, after which it will crash with " Unable to get property 'data' of undefined or null reference (http://localhost:5601/bundles/kibana.bundle.js?v=15443:111) ". Browser IE11.

I did a search and found this discussion, where it states that this bug is present in 5.6.0 and was introduced in 5.5.2:

Last autumm ElasticSearch had a huge error in setting up Garbage collection on Windows 4.x , which resulted in a ton of crashes . Highly embarrasing. Lack of testing.

Which lead me to my question. Why are you releasing this kind of poor quality? I absolutely love this product and want to make it known and accepted in the Enterprise world, where you are competing with Splunk and Arcsight and others, but these kind of bugs are totally unacceptable. I find that upgrading an Elastic cluster to a new version is always a risky business, wondering what is broken this time.

Before you release any kind of new version of Kibana , you should be doing quality control and making sure , it works in all major browsers. An acceptance test would have caught this error in a heartbeat.

I wonder if you are sacrificing build quality in order to pump out new versions faster than you are able to keep up with.

Again , I love this product and it has saved me many times , but it is highly frustrating to have new errors to cope with every single time I upgrade.

Best regards
Kim

2

I'm sorry that this bug impacted you, and it's really unfortunate that it ever made it into a released build. We must do better.

We do have multiple layers of QA in place for stack releases, including peer review on changes before they get merged, unit tests and integration tests that are each run on every commit, and manual test plans performed by a dedicated QA team on build candidates leading up to a release. Even with all of these things in place, we're not entirely satisfied with the test coverage today, and we're going to continue to expand and improve upon our QA mechanisms every single day.

Cross-browser problems are generally pretty rare for Kibana since we compile the JavaScript down to a stable version of the language spec that all of our browsers support, but occasionally we discover compatibility issues that arise when we're using browser APIs that may differ between browsers. Expanding our test coverage across all supported browsers is an essential and ongoing effort.

For folks that require stability above all else (which may or may not be you), we do provide a legacy stable version of the products. Our current legacy stable version is 4.6.x, and it will be maintained up until the GA release of 6.0.0.

1 Like
3

@Kim-Kruse-Hansen out of interest, are you an open source user or do you have a subscription with us?

4

Hi Mark,

Unfortunately , I am still currently open-source user.

Regards
Kim

5

Hi Court,

I appreciate your answer. I am sure you are working hard on improving things and I do appreciate, its a very complicated scenario.

However , I fail to understand , why you dont release a fix to this when you become aware of this or pull the version from the downloads page or simply put out a general warning (like known problems) about this? If you have done any of this, I am sorry, but I have not seen it.

Keep up the great work
Kim

6

I think a message about this as a known issue in 5.5.2 is a great idea, so people don't upgrade Kibana if they need to use IE until we get the fix released.

Every release we make takes time to go through all of the various testing, both automated and manual, so we're turning around a release with the fix as quickly as we can. That shouldn't stop us from letting people know so they don't get surprised when they upgrade, though.

7

Hi Court,

A known issues section on the download page of your respective components would be a great help. Not just for this specific bug, but as a general way of communicating with your users in the future.

Best regards
Kim

8

I pushed a change to the 5.5.2 release notes to include a "known issues" section with this particular regression highlighted. I'll look into the downloads page change as well, but I wanted to get something up to indicate the issue for folks asap.

Thanks again for your feedback here.

9

Hi

Great, thanks for listening and responding so fast

Regards
Kim

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.