# ECK: Added nodes and now queries are unable to find stored scripts **URL:** \ **Category:** Elasticsearch\ **Created:** [July 21, 2021, 6:15pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301 "2021-07-21T18:15:10Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [July 21, 2021, 6:15pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/1 "2021-07-21T18:15:10Z")
ES: 6.8.13 We're in the process of moving from manually maintained ES clusters to Elastic Cloud on Kubernetes (ECK). In our first test environment (a single node cluster), we added 3 ECK nodes. Afterward, all our searches that engaged a stored script started failing. ```auto Status = 404 { "error": { "root_cause": [ { "type": "resource_not_found_exception", "reason": "unable to find script [xxx] in cluster state" ... ``` As I went to list the scripts, I discovered there were none there any more. `GET _cluster/state/metadata?pretty&filter_path=**.stored_scripts` I found this [elastic discussion article](https://discuss.elastic.co/t/elasticsearch-restart-causes-error-unable-to-find-script-q-checkitem-in-cluster-state/223000) where someone reported something similar but they never replied to @spinscale question. I assume that stored scripts are replicated when a new node is added, correct? I have since tested adding some of our scripts back and those show up in the list and our queries that rely on them work again. So, where could they have gone? 🤷‍♂️ I'm worried we're missing something and, if not, I certainly don't want to have to reapply all our scripts in every environment where we move to ECK. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 22, 2021, 1:38pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/2 "2021-07-22T13:38:38Z")
Scripts are stored in the cluster state indeed. When you moved over to ECK, how did you ensure that the scripts were moved over was well? As it is part of the cluster state, every node has it. Can you share your migration path? Maybe that sheds some light.. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [July 22, 2021, 6:56pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/3 "2021-07-22T18:56:21Z")
@spinscale Thanks for the response. Sure, all we did was just add the new ECK nodes to the existing single node cluster in preparation to retire the original (which we still haven't done). All of the indexes replicated just fine. The scripts were a different story as I described above so I was figuring that we did something wrong or made some assumptions about how stored scripts are stored and\or propagated. Sounds like they should also have propagated to the new nodes just like the index data did. 🤔 ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 23, 2021, 8:03am UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/4 "2021-07-23T08:03:28Z")
That sounds about right. If you run ```auto GET _cluster/state/metadata?filter_path=metadata.stored_scripts ``` do you see your stored scripts? ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [July 24, 2021, 12:14am UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/5 "2021-07-24T00:14:39Z")
Nope. > [@Thomas\_Doman](#): > > As I went to list the scripts, I discovered there were none there any more. > `GET _cluster/state/metadata?pretty&filter_path=**.stored_scripts` ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 26, 2021, 7:20am UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/6 "2021-07-26T07:20:45Z")
Hm, running out of ideas a little. Can you check the logs, if there is any error or exception listed? Also, from which version to which version did you run the upgrade? Which JVM versions are you using? Thanks! ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [July 26, 2021, 5:49pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/7 "2021-07-26T17:49:58Z")
> Can you share your migration path? Maybe that sheds some light.. We mostly followed this document [Remote clusters | Elastic Cloud on Kubernetes [master] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-remote-clusters.html) . Our scenario is the "Connect from an Elasticsearch cluster running outside the Kubernetes cluster". > Also, from which version to which version did you run the upgrade? We didn't upgrade. Both clusters were and still are running `6.8.13` . We're using ECK 1.6 if that's relevant to troubleshooting. > Which JVM versions are you using? I believe this is the information you're looking for: $JAVA\_HOME = `/opt/jdk-15+36` [root@esdev-es-all-128gi-0 bin]# /opt/jdk-15+36/bin/java -version openjdk version "15" 2020-09-15 OpenJDK Runtime Environment AdoptOpenJDK (build 15+36) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15+36, mixed mode, sharing) ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 27, 2021, 9:02am UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/8 "2021-07-27T09:02:38Z")
aaaaaah. That sheds some light and explains the behaviour. Glad we found it. Side note: Make sure to select the documentation that fits to your ECK version on the right navigation bar. You've been reading the one for the `master` branch. So, a remote cluster is not the same than a node joining the cluster. A remote cluster allows to connect two clusters together, while they are still two independent clusters. This is useful for cross cluster search for cross cluster replication. See [Remote clusters | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/modules-remote-clusters.html) This is the reason why the scripts are not synced up, as you still have two fully independent clusters. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [July 27, 2021, 6:39pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/9 "2021-07-27T18:39:36Z")
@spinscale Good call out, we should have made sure we were referencing the documentation for ES 6.8. Speaking of ES 6.8, as I've mentioned, that's the version we're using. So, that precludes cross-cluster replication as the cause of scripts not replicating, right? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [July 28, 2021, 8:00am UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/10 "2021-07-28T08:00:42Z")
Exactly.. remote clusters do initiate a permanent connection to another cluster, but it is 'only' usable for CCS and CCR and does not behave like a node joining a cluster and retrieving the cluster state and using it locally. ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [August 1, 2021, 5:23pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/11 "2021-08-01T17:23:00Z")
Sorry, I'm saying , we're not using a version of ES which supports CCR so that cannot be what caused the issue we experienced. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [August 2, 2021, 7:30am UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/12 "2021-08-02T07:30:57Z")
You are using the remote cluster functionality, which will not copy any scripts over into the cluster state. The remote cluster functionality is basically the foundation for CCS and CCR, but does not imply you use those, it's just the way you connect to a remote cluster. I am not sure I fully understood your last post.. please explain 🙂 We found the issue here it seems, but you want to do achieve something else, right? ---
**Author:** ![Thomas\_Doman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_doman/32/11409_2.png) [@Thomas\_Doman](https://discuss.elastic.co/u/Thomas_Doman)\ **Post date:** [August 2, 2021, 11:28pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/13 "2021-08-02T23:28:18Z")
I see, since you were referring to CCS and CCR which are not part of the version of ES we're using, I assumed that couldn't be the problem. It sounds like the "remote cluster" functionality _ **is** _ available in ES 6.8 and is leveraged by ECK via the method we used. Thanks for the explanation! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2021, 11:28pm UTC](https://discuss.elastic.co/t/eck-added-nodes-and-now-queries-are-unable-to-find-stored-scripts/279301/14 "2021-08-30T23:28:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.