# When indices only exist on remote cluster: datafeed \[xxxx\] cannot retrieve data because no index matches datafeed's indices **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [March 19, 2024, 11:42am UTC](https://discuss.elastic.co/t/when-indices-only-exist-on-remote-cluster-datafeed-xxxx-cannot-retrieve-data-because-no-index-matches-datafeeds-indices/355718 "2024-03-19T11:42:01Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![sparrowt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sparrowt/32/47163_2.png) [@sparrowt](https://discuss.elastic.co/u/sparrowt)\ **Post date:** [March 19, 2024, 11:42am UTC](https://discuss.elastic.co/t/when-indices-only-exist-on-remote-cluster-datafeed-xxxx-cannot-retrieve-data-because-no-index-matches-datafeeds-indices/355718/1 "2024-03-19T11:42:01Z")
I am getting the following error when trying to start the datafeed of an ML Anomaly Detection Job: ```plaintext mydatafeed failed to start datafeed [mydatafeed] cannot retrieve data because no index matches datafeed's indices [foo.bar*, *:foo.bar*] ``` 'See the full error' shows this: ```js { "error": { "root_cause": [ { "type": "status_exception", "reason": "datafeed [mydatafeed] cannot retrieve data because no index matches datafeed's indices [foo.bar*, *:foo.bar*]" } ], "type": "status_exception", "reason": "datafeed [mydatafeed] cannot retrieve data because no index matches datafeed's indices [foo.bar*, *:foo.bar*]" }, "status": 400 } ``` The setup is as follows, on v8.7.0 - one elastic cluster which we'll call the "primary" with Kibana and ML node(s) - other linked trusted elastic clusters used for Cross Cluster Search (CCS) I have several machine learning jobs running fine, but am trying to add a new one for a different set of `indices`. It seems to me that the crucial difference with this new one is that _currently_ there are no indices/datastreams matching `foo.bar` on the "primary" elastic cluster. These are hopefully the relevant bits of the `datafeed_config` in the ML job: ```js "indices_options": { "expand_wildcards": [ "open" ], "ignore_unavailable": false, "allow_no_indices": true, "ignore_throttled": true }, "query": { ... }, "indices": [ "foo.bar*" "*:foo.bar*" ], "scroll_size": 1000, "delayed_data_check_config": { "enabled": true } ``` I've tried removing `foo.bar*` from the list of `indices` so that the job config is just like this: (although I want to keep it there because new data _might_ appear on the primary in future) ```js "indices": [ "*:foo.bar*" ], ``` but it fails in the same way. My issue seems _similar_ to [[ML] Datafeed fails on missing indices, even with allow\_no\_indices set to true · Issue #62404 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/62404) (and I do have `allow_no_indices` as `true`) but the difference here is that we have **remote clusters** involved, so there _are_ actually indices available which the datafeed could start consuming. The error I'm receiving seems to come from this code [elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractorFactory.java at v8.7.0 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/blob/v8.7.0/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/scroll/ScrollDataExtractorFactory.java#L107) I found an interesting comment here [elasticsearch/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedNodeSelector.java at v8.7.0 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/blob/v8.7.0/x-pack/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/DatafeedNodeSelector.java#L183) which suggests there is an intention to succeed in the case of remote indices. However in my case it seems to have got past this point because a node has been allocated: if I look in Kibana at `/app/ml/jobs` expand the job row and look at 'Job messages' it says: > Opening job on node [instance-0000000090] so it is failing at a later stage when it tries to actually consume the datafeed. Is this a bug that I should report at [GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine](https://github.com/elastic/elasticsearch/) do you think? I noticed the code that produces the error is looking at a `FieldCapabilitiesResponse` so perhaps in fact there is an (implicit) requirement that the cluster where the ML job runs has _at least 1_ index matching the `indices` of the datafeed? (so that [Field capabilities API | Elasticsearch Guide [8.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-field-caps.html) can return relevant data) If so is this / should it be documented somewhere? Is there any way around this given the setup described above? I don't want ML nodes on the remote clusters - even if I did choose one remote where foo.bar indices are present, it would only see its own indices, not those from _other_ remotes. ---
**Author:** ![sparrowt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sparrowt/32/47163_2.png) [@sparrowt](https://discuss.elastic.co/u/sparrowt)\ **Post date:** [March 20, 2024, 4:12pm UTC](https://discuss.elastic.co/t/when-indices-only-exist-on-remote-cluster-datafeed-xxxx-cannot-retrieve-data-because-no-index-matches-datafeeds-indices/355718/2 "2024-03-20T16:12:15Z")
In the meantime I've managed to work around this by posting a single fake document on the "primary" elastic cluster, so that the datastream exists and thus there is at least 1 matching index and the datafeed will start. ```js POST /foo.bar.datastream/_doc/ { "@timestamp": "2024-03-20T12:00:00.000Z", "log.level": "INFO", "log.logger": "not.a.real.logger", "log.origin.function": "not.a.real.function", "message": "This is not a real log message, it is posted from Kibana Dev Tools purely in order to ensure that a foo.bar* index exists so that the Anomaly Detection job will start" } ``` I included a minimal set of fields which are all the ones referenced by my ML job (e.g. in the datafeed query / analysis\_config / influencers) on the assumption that the field capabilities check was probably going to want to check on at least some of these. This is of course not at all ideal - it requires polluting the production data with this random log line, just in order to get the datafeed started. And if I want to create a new job in future (and this magic log line has been aged out & deleted by the ILM policies governing foo.bar) then I'll have to do this again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 17, 2024, 4:12pm UTC](https://discuss.elastic.co/t/when-indices-only-exist-on-remote-cluster-datafeed-xxxx-cannot-retrieve-data-because-no-index-matches-datafeeds-indices/355718/3 "2024-04-17T16:12:34Z")
Closed on April 17, 2024, 4:12pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.