How to delete elasticsearch cross cluster search seed 5.5.1-1

[chartley42]

I'm running elasticsearch 5.5.1-1 and x-pack for monitoring.

Elasticsearch's documentation say's I should be able to add the following code to my elasticsearch.yml file for cross cluster search seeding:

search:
    remote:
        cluster_one: 
            seeds: 1.1.1.1:9300
        cluster_two: 
            seeds: 2.2.2.2:9300

Well that didn't work so I used the API as follows:

PUT _cluster/settings
{
  "persistent": {
    "search": {
      "remote": {
        "cluster_one": {
          "seeds": [
            "1.1.1.1:9300"
          ]
        },
        "cluster_two": {
          "seeds": [
            "2.2.2.2:9300"
          ]
        }
      }
    }
  }
}

That worked fine however I needed to delete cluster_one until it was upgraded to 5.5.1-1 which according to elasticsearch's docs say's should be completed as follows:

PUT _cluster/settings
{
  "persistent": {
    "search": {
      "remote": {
        "cluster_one": {
          "seeds": null 
        }
      }
    }
  }
}

It appears to takeas I get the acknowledgement:

    {
      "acknowledged" : true,
      "persistent" : { },
      "transient" : { }
    }

However if I curl the cluster settings I still see both nodes as follows:

   {
      "persistent" : {
        "search" : {
          "remote" : {
            "cluster_one" : {
              "seeds" : [
                "1.1.1.1:9300"
              ]
            },
            "cluster_two" : {
              "seeds" : [
                "2.2.2.2:9300"
              ]
            }
          }
        }
      },
      "transient" : { }
    }

I wouldn't think I needed to restart elasticsearch after the API call but I tried that to no avail as well.

I also played with quoting/capitalizing null and got this return message which leads me to believe null is a valid value:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "json_parse_exception",
        "reason" : "Unrecognized token 'NULL': was expecting 'null', 'true', 'false' or NaN\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@6053e99; line: 7, column: 25]"
      }

[chartley42]

So I performed a downgrade/upgrade in hopes it would remove the seeds.... it did not.

I performed an uninstall/reinstall with the same hope..... it still did not remove the seeds.

I used locate to find every file in the file system and grepped each file for the IP and found the only file with the IP in it:

/var/lib/elasticsearch/nodes/0/_state/global-14.st

I rm'ed that file and restart elasticsearch and the seed settings were finally gone.

[warkolm (Mark Walkom)]

@chartley42

I rm’ed that file and restart elasticsearch and the seed settings were finally gone.

That is not good, playing with state files on disk is likely to cause other problems.

@chartley42

Well that didn’t work

Why not? Where there errors? What were they?

[chartley42]

Well the global state file rebuilt and thus far all seems well, and I'd rather not have done it either, but as the API wouldn't allow me to remove the seeds I had little choice.

As for the errors with the cluster seed settings being set in the elasticsearch.yml file the first seed was not upgraded yet but I expected it to not connect to the first cluster but startup fine and connect to the second cluster. It never attempted to connect to the second cluster and kibana went red saying couldn't connect... elasticsearch plugin red... etc...

I rearranged the order and even removed the other seed from the yaml to no avail. I do not have any logs from that period as I kept tinkering until I got the API to take the seed settings. And only posted online when I couldn't remove the seed that was not upgraded.