Machine learning problems in Elastic 6

@vb4t - sorry to hear that your upgrade experience was difficult.

Machine Learning is supported and tested in Elastic 6.0, but unfortunately, there are some known issues with ML datafeed upgrades from 5.6 (Elasticsearch 5.6.3 upgrade to 6.0 Node will not start "failed to read local state" etc.)

The primary issue is when a query used in an ML datafeed created in version 5.x uses search DSL syntax that was deprecated in version 5.x and removed in version 6.0. Any of the removed query syntax listed in https://www.elastic.co/guide/en/elasticsearch/reference/6.0/breaking_60_search_changes.html could cause the problem.

A workaround is to resolve issues with the datafeeds in 5.x jobs PRIOR to upgrade (either via https://www.elastic.co/guide/en/elasticsearch/reference/5.6/ml-update-datafeed.html or via the UI), but a better fix is to upgrade first to 5.6.5 and then use the upgrade assistant.

The upgrade assistant in 5.6.5 will perform additional checks on ML datafeeds that tell users which of their datafeeds will suffer from this problem.

Therefore, customers running ML are advised to upgrade to version 5.6.5 before running the upgrade assistant. This should have hopefully avoided some of your issues.

In terms of your other comments, the JSON in the ML UI is not designed to be cut and pasted into new jobs. It contains configuration and information such as data_counts which are local to that job. Currently, the ML API (https://www.elastic.co/guide/en/x-pack/current/ml-api-quickref.html) is better suited to low level job creation via JSON.