Machine learning problems in Elastic 6

[vb4t]

Hello, is Machine Learning still supported and tested feature in Elastic 6? I have big problems after upgrading from 5.6 to 6.0:

1. Before upgrade I had to remove all ML jobs. Upgrade did not work with machine learning jobs in place.
2. Recreating machine learning jobs with saved JSONs does not work. It simply can not parse it. Misc errors.
3. Creation of jobs with "create a new job" fails if using saved searches from ES 5.0. I found it fails if it contains a space between FIELD and VALUE (field: value) ... btw. these searhes works in discover
4. Creation of a new job often fails with 500 internal server error - usually if there is a lot of data or just randomly.
5. Cloning of the existing job does not work - unable to parse description, unable to parse .... errors.

[stevedodson]

@vb4t - sorry to hear that your upgrade experience was difficult.

Machine Learning is supported and tested in Elastic 6.0, but unfortunately, there are some known issues with ML datafeed upgrades from 5.6 (Elasticsearch 5.6.3 upgrade to 6.0 Node will not start "failed to read local state" etc.)

The primary issue is when a query used in an ML datafeed created in version 5.x uses search DSL syntax that was deprecated in version 5.x and removed in version 6.0. Any of the removed query syntax listed in https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_search_changes.html could cause the problem.

A workaround is to resolve issues with the datafeeds in 5.x jobs PRIOR to upgrade (either via https://www.elastic.co/guide/en/elasticsearch/reference/5.6/ml-update-datafeed.html or via the UI), but a better fix is to upgrade first to 5.6.5 and then use the upgrade assistant.

The upgrade assistant in 5.6.5 will perform additional checks on ML datafeeds that tell users which of their datafeeds will suffer from this problem.

Therefore, customers running ML are advised to upgrade to version 5.6.5 before running the upgrade assistant. This should have hopefully avoided some of your issues.

In terms of your other comments, the JSON in the ML UI is not designed to be cut and pasted into new jobs. It contains configuration and information such as data_counts which are local to that job. Currently, the ML API (https://www.elastic.co/guide/en/x-pack/current/ml-api-quickref.html) is better suited to low level job creation via JSON.

[vb4t]

Currently I restarted cluster and one of the nodes could not connect to cluster - some machine learning error about reading stat file. Currently I have 1 ML job created in 6.0.1 so this problem is not only related to upgrading. I had to remove data dir for that data node.