Hi,
we are running Elasticsearch 6.5.4 and we started to use cross cluster search since 6.3. Our settings were using the search.remote section in the cluster settings and since 6.5 we started using the new preferred cluster.remote section.
Recently I realized that we have both sections in our cluster settings with both configuring same clusters, e.g.:
persistent.search.remote.cluster1.seeds : ["node1", "node2"]
and
persistent.cluster.remote.cluster1.seeds : ["node10", "node12"]
Note that node1 and node2 are no longer part of cluster1.
After a rolling-restart of this cluster we noticed that search requests relyng on CCS failed with: remote_transport_exception: [error while communicating with remote cluster [cluster1]].
I tried removing the persistent.search.remote section but it failed saying persistent setting [search.remote], not recognized.
I tried to empty the seeds of persistent.search.remote.cluster1 but it emptied the seeds of persistent.cluster.remote.cluster1.
I re-added proper seeds (node10 and node12 here) to persistent.search.remote.cluster1 and the errors stopped but it only affected persistent.cluster.remote.cluster1 leaving persistent.search.remote.cluster1 with the old nodes.
In the end the settings remain the same as when the errors occurred:
persistent.search.remote.cluster1.seeds : ["node1", "node2"]
persistent.cluster.remote.cluster1.seeds : ["node10", "node12"]
Could there be some loading inconsistency that could explain why the same settings caused CCS errors but that the errors disappeared when some of the settings were re-imported again?
Is there a way to purge the old persistent.search settings with 6.5.4?
I suspect that this might be related to:
- Generalize search.remote settings to cluster.remote by jasontedor · Pull Request #33413 · elastic/elasticsearch · GitHub
- and Fix remote cluster seeds fallback by jasontedor · Pull Request #34090 · elastic/elasticsearch · GitHub
But looking at the later (which seems to fix some issues related to what we see) I see that it might fail when there's a duplicated cluster in both cluster.remote and search.remote section and thus I worry that if we don't manage to clean up this old search.remote section we might hit a wall when upgrading to newer versions of Elasticsearch.
Thank you!
David.
)