# ES starts, but Cross Cluster Search shows local data node disconnected **URL:** \ **Category:** Elasticsearch\ **Created:** [September 5, 2018, 7:43pm UTC](https://discuss.elastic.co/t/es-starts-but-cross-cluster-search-shows-local-data-node-disconnected/147466 "2018-09-05T19:43:07Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![mtowle](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@mtowle](https://discuss.elastic.co/u/mtowle)\ **Post date:** [September 5, 2018, 7:43pm UTC](https://discuss.elastic.co/t/es-starts-but-cross-cluster-search-shows-local-data-node-disconnected/147466/1 "2018-09-05T19:43:07Z")
I have two lab ES 6.2.2 data nodes, and one data node also acts as a Cross Cluster Search node. When I start the CCS node for the first time with a script that starts up ES upon reboot, curl shows the node is connected, but when I dig a little deeper looking at the CCS connectivity, it shows the local node being disconnected. If I restart ES, everything works fine. Here's the initial curl output following a reboot: # curl http://"\":9200 { "name" : "node2", "cluster\_name" : "cluster2", "cluster\_uuid" : "oaTc47YtQ\_WFip5zoRvzag", "version" : { "number" : "6.2.2", "build\_hash" : "10b1edd", "build\_date" : "2018-02-16T19:01:30.685723Z", "build\_snapshot" : false, "lucene\_version" : "7.2.1", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } But when I look at the CCS info, here's what I see: # curl -XGET 'localhost:9200/\_remote/info?pretty' { "cluster2" : { "seeds" : [], "http\_addresses" : [], "connected" : false, "num\_nodes\_connected" : 0, "max\_connections\_per\_cluster" : 3, "initial\_connect\_timeout" : "30s", "skip\_unavailable" : false }, "cluster1" : { "seeds" : [ "[\]:9300" ], "http\_addresses" : [ "[\]:9200" ], "connected" : true, "num\_nodes\_connected" : 1, "max\_connections\_per\_cluster" : 3, "initial\_connect\_timeout" : "30s", "skip\_unavailable" : false } } Here's my elasticsearch.yml file on the CCS node, with all commented out lines removed for brevity: cluster.name: cluster2 node.name: node2 bootstrap.memory\_lock: true network.bind\_host: 0.0.0.0 network.publish\_host: _eth0:ipv6_ network.tcp.keep\_alive: true bootstrap.system\_call\_filter: false search.remote.cluster1.seeds: "\:9300" search.remote.cluster2.seeds: "\:9300" Has anyone seen this behavior before, and if so, what did you do to solve it? Thanks in advance! ---
**Author:** ![mtowle](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@mtowle](https://discuss.elastic.co/u/mtowle)\ **Post date:** [September 7, 2018, 12:17pm UTC](https://discuss.elastic.co/t/es-starts-but-cross-cluster-search-shows-local-data-node-disconnected/147466/2 "2018-09-07T12:17:51Z")
I looked this morning, and while both my nodes showed being connected, remote node1's CCS status showed a disconnection exception. The node\_disconnected\_exception took several minutes to show up after issuing the curl command. Here's what I'm seeing. These commands were all being issued on the CCS node2. Any ideas on how to fix this would be much appreciated. Please note that I don't have any data being indexed yet. # curl [http://localhost:9200](http://localhost:9200) { "name" : "node2", "cluster\_name" : "cluster2", "cluster\_uuid" : "oaTc47YtQ\_WFip5zoRvzag", "version" : { "number" : "6.2.2", "build\_hash" : "10b1edd", "build\_date" : "2018-02-16T19:01:30.685723Z", "build\_snapshot" : false, "lucene\_version" : "7.2.1", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } # curl [http://node1:9200](http://node1:9200) { "name" : "node1", "cluster\_name" : "cluster1", "cluster\_uuid" : "IbNavfZmRLazxMjYsQ2d3w", "version" : { "number" : "6.2.2", "build\_hash" : "10b1edd", "build\_date" : "2018-02-16T19:01:30.685723Z", "build\_snapshot" : false, "lucene\_version" : "7.2.1", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } # curl -XGET 'localhost:9200/\_remote/info?pretty' { "error" : { "root\_cause" : [ { "type" : "node\_disconnected\_exception", "reason" : "[node1][[\]:9300][cluster:monitor/nodes/info] disconnected" } ], "type" : "node\_disconnected\_exception", "reason" : "[node1][[\]:9300][cluster:monitor/nodes/info] disconnected" }, "status" : 500 } # curl [http://node1:9200](http://node1:9200) { "name" : "node1", "cluster\_name" : "cluster1", "cluster\_uuid" : "IbNavfZmRLazxMjYsQ2d3w", "version" : { "number" : "6.2.2", "build\_hash" : "10b1edd", "build\_date" : "2018-02-16T19:01:30.685723Z", "build\_snapshot" : false, "lucene\_version" : "7.2.1", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } # curl -XGET 'localhost:9200/\_remote/info?pretty' { "cluster2" : { "seeds" : [ "[\]:9300" ], "http\_addresses" : [ "[\]:9200" ], "connected" : true, "num\_nodes\_connected" : 1, "max\_connections\_per\_cluster" : 3, "initial\_connect\_timeout" : "30s", "skip\_unavailable" : false }, "cluster1" : { "seeds" : [ "[\]:9300" ], "http\_addresses" : [ "[\]:9200" ], "connected" : true, "num\_nodes\_connected" : 1, "max\_connections\_per\_cluster" : 3, "initial\_connect\_timeout" : "30s", "skip\_unavailable" : false } } ---
**Author:** ![mtowle](https://avatars.discourse-cdn.com/v4/letter/m/e495f1/32.png) [@mtowle](https://discuss.elastic.co/u/mtowle)\ **Post date:** [September 13, 2018, 3:37pm UTC](https://discuss.elastic.co/t/es-starts-but-cross-cluster-search-shows-local-data-node-disconnected/147466/3 "2018-09-13T15:37:34Z")
Looks like this was related to my JVM heap settings. They were set in the jvm.options file to the defaults of -Xms1g and -Xmx1g. My servers have 49GB memory, so I changed the settings to -Xms24g and -Xmx24g and rebooted. ES started, and CCS shows both clusters connected. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 11, 2018, 3:37pm UTC](https://discuss.elastic.co/t/es-starts-but-cross-cluster-search-shows-local-data-node-disconnected/147466/4 "2018-10-11T15:37:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.