# How to create an automated solution for Instance Migration in the case of Allocator Faliures?

**URL:** <https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369>\
**Category:** Elastic Cloud Enterprise (ECE)\
**Tags:** migration\
**Created:** [March 22, 2022, 4:59pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369 "2022-03-22T16:59:21Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 22, 2022, 4:59pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/1 "2022-03-22T16:59:21Z")

</div>

I've found this for the manual method: [Move nodes or instances from allocators | Elastic Cloud Enterprise Reference [3.1] | Elastic](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-move-nodes.html)

I looked at the APIs [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/rest-apis.html) but couldn't find anything relevant to my situation.

I'm new to Elastic and am not sure were to start looking so would appreciate any assistance in pointing me to the right direction.

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 23, 2022, 7:52pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/2 "2022-03-23T19:52:51Z")

</div>

I did manage to find some docs that could help, I could use watcher to create an alert which could then trigger the migration

- Watcher

- Regarding the action the watcher would have to take

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 5:02pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/3 "2022-03-28T17:02:59Z")

</div>

I'm currently working on testing the sensing part of the watcher using the following query but I'm having trouble accessing the API using my API key (not sure how to setup the key, I've only seen examples for user/password setup).

```auto
{
  "trigger": {
    "schedule": {
      "interval": "2m"
    }
  },
  "input": {
    "http" : {
      "request" : {
        "scheme": "http",
        "host" : "$COORDINATOR_HOST",
        "port" : 12443,
        "path" : "/api/v1/platform/infrastructure/allocators",
        "method": "get",
        "auth":{
                  "Authorization": "ApiKey $ECE_API_KEY"
        }
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "{{ctx}}"
      }
    }
  }
}

```

Because without the authorization I get no response from the server:

> "error": {  
> "root\_cause": [  
> {  
> "type": "no\_http\_response\_exception",  
> "reason": "$COORDINATOR\_HOST:12443 failed to respond"  
> }

But I know that it can send a response because I've tested it on a terminal and got the expected response:

```auto
curl -k -X GET -H "Authorization: ApiKey $ECE_API_KEY" https://$COORDINATOR_HOST:12443/api/v1/platform/infrastructure/allocators

```

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 5:17pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/4 "2022-03-28T17:17:19Z")

</div>

> [@Apprentice](#):
>
> "$COORDINATOR\_HOST:12443 failed to respond

Based on that error it looks more likely that it's not actually reaching the correct URL than anything to do with the API key (which I would expect a 401 response, not `no_http_response_exception`). If you put the hardcoded value for the ECE host instead of `$COORDINATOR_HOST` does it work?

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 5:18pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/5 "2022-03-28T17:18:44Z")

</div>

I have hardcoded in the ECE host value, I'm not actually pointing to any variable.

I'm confused because I use the same information for the terminal command as I mentioned previously and it worked fine.

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 5:20pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/6 "2022-03-28T17:20:31Z")

</div>

> [@Apprentice](#):
>
> `12443`

Hmm I just noticed, you are using the HTTPS port 12443 but a scheme of `http`. Can you change the scheme to `https` or the port to `12400`?

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 5:22pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/7 "2022-03-28T17:22:27Z")

</div>

changed the scheme to `https`, but now I'm getting the following error:

> "error": {  
> "root\_cause": [  
> {  
> "type": "s\_s\_l\_handshake\_exception",  
> "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"  
> }

I should mention that I am trying to use a watcher to pull info from the ECE API, will that cause any issues?

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 6:34pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/8 "2022-03-28T18:34:05Z")

</div>

That looks like the JVM doesn't know about your SSL certificate. I think you might need to use one from a known public CA. Maybe try for now using `http` and port `12400` while testing out the overall workflow? I don't see any reason why this should have issues hitting the ECE API if everything is configured correctly.

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 6:50pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/9 "2022-03-28T18:50:59Z")

</div>

Using `http` and port `12400` gives me the following error:

> "type": "http",  
> "status": "failure",  
> "error": {  
> "root\_cause": [  
> {  
> "type": "http\_host\_connect\_exception",  
> "reason": "Connect to [hostname] failed: Connection refused"  
> }  
> ],  
> "type": "http\_host\_connect\_exception",  
> "reason": "Connect to [hostname] failed: Connection refused",  
> "caused\_by": {  
> "type": "connect\_exception",  
> "reason": "Connection refused"  
> }  
> },

I noticed the `curl` command specified in the docs uses the `-k` option, _"this option makes curl skip the verification step and proceed without checking"_. Is there any similar option I can use in the watcher?

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 6:57pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/10 "2022-03-28T18:57:27Z")

</div>

> [@Apprentice](#):
>
> Connect to [hostname] failed: Connection refused

This is a general networking error, I guess the URL isn't valid (maybe you are using a load balancer in front of ECE and not exposing the regular http port?). Are you able to use curl to http://$COORDINATOR\_HOST:12400"? If that works I'd expect the watcher to work as well (no need for -k here since there is no SSL certificates).

I'm not sure if watcher allows you to bypass SSL certificate warnings but I'd imagine it doesn't as this is insecure, you should generally either use regular HTTP or else HTTPS with a valid certificate.

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 7:12pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/11 "2022-03-28T19:12:46Z")

</div>

Thats why I find it so odd, I'm able to use Curl to get the information but not the watcher. The exact curl command I used is:

```auto
curl -k -X GET -H "Authorization: ApiKey [key value here]" https://[host url here]:12443/api/v1/platform/infrastructure/allocators

```

And the watcher query is as follows:

```auto
{
  "trigger": {
    "schedule": {
      "interval": "2m"
    }
  },
  "input": {
    "http" : {
      "request" : {
        "scheme": "http",
        "host" : "[host url here]",
        "port" : 12400,
        "path" : "/api/v1/platform/infrastructure/allocators",
        "method": "get",
        "auth":{
          "Authorization": "ApiKey [key value here]"
        }
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "{{ctx}}"
      }
    }
  }
}

```

This gives me the following error:

```auto
      "status": "failure",
      "error": {
        "root_cause": [
          {
            "type": "http_host_connect_exception",
            "reason": "Connect to [host url here]:12400 failed: Connection refused"
          }
        ],
        "type": "http_host_connect_exception",
        "reason": "Connect to [host url here]:12400 failed: Connection refused",
        "caused_by": {
          "type": "connect_exception",
          "reason": "Connection refused"
        }
      },

```

When I change the scheme to `https` and port to `12443`, I get the certs error:

```auto
"type": "http",
      "status": "failure",
      "error": {
        "root_cause": [
          {
            "type": "s_s_l_handshake_exception",
            "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
          }
        ],
        "type": "s_s_l_handshake_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "validator_exception",
          "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
          "caused_by": {
            "type": "sun_cert_path_builder_exception",
            "reason": "unable to find valid certification path to requested target"
          }
        }
      },

```

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 7:16pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/12 "2022-03-28T19:16:14Z")

</div>

I'm simulating the watcher in _Cloud UI \> Stack Management \> Watcher \> Create_, so it's not even like I'm trying to make a request from outside

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 7:42pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/13 "2022-03-28T19:42:46Z")

</div>

> [@Apprentice](#):
>
> Thats why I find it so odd, I'm able to use Curl to get the information but not the watcher. The exact curl command I used is:

This makes sense though, the `-k` option is telling curl you don't care about the SSL certificate not being valid. I would test anything in curl that you want to run in watcher without using `-k`.

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 7:43pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/14 "2022-03-28T19:43:19Z")

</div>

So watcher doesn't have anything similar to ignore verification?

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 7:43pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/15 "2022-03-28T19:43:57Z")

</div>

Not that I'm aware of (I'm on the ECE team though so I can't really speak as a Watcher expert).

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 8:02pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/17 "2022-03-28T20:02:33Z")

</div>

I've found [xpack.http.ssl.verification\_mode](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/notification-settings.html#ssl-notification-settings) parameter to help me ignore the certs issue to allow testing for now and I think I am able to reach the ECE API now.

However it seems that the way I am sending the ECE api key via the get request is incorrect and I'm not sure of the correct format as all examples in the doc only show me a username/password example

> [@Apprentice](#):
>
> ```auto
> "auth":{
> "Authorization": "ApiKey [key value here]"
> }
> 
> ```

I am getting the following error:

```auto
errors=[{code=root.unauthenticated, message=The supplied authentication is invalid}]}

```

Full Response to the action:

```auto
"logged_text": "{metadata={name=allocator_monitoring, xpack={type=json}}, watch_id=_inlined_, payload={_headers={x-request-id=[*alpha-numeric stuff here*], date=[Mon, 28 Mar 2022 19:58:06 GMT], server=[*alpha-numeric stuff here*], transfer-encoding=[chunked], vary=[Accept-Encoding], x-frame-options=[SAMEORIGIN], www-authenticate=[Bearer realm=\"found-adminconsole\"], x-download-options=[noopen], strict-transport-security=[max-age=3600; includeSubDomains], set-cookie=[ec-session=; Max-Age=0; Path=/; Secure; HttpOnly], content-security-policy=[default-src 'none';script-src 'self';worker-src 'self' blob:;connect-src 'self' https://telemetry.elastic.co;img-src 'self' data:;style-src 'self' 'unsafe-inline';manifest-src 'self';font-src 'self';frame-src 'self'], x-content-type-options=[nosniff], x-xss-protection=[1;mode=block], content-type=[application/json], connection=[Keep-Alive], x-ui-tag=[indeterminate], x-cloud-error-codes=[root.unauthenticated], cache-control=[max-age=0, no-cache, no-store, must-revalidate]}, _status_code=401, errors=[{code=root.unauthenticated, message=The supplied authentication is invalid}]}, id=_inlined__*alpha-numeric stuff here*-2022-03-28T19:58:06.507945975Z, trigger={triggered_time=2022-03-28T19:58:06.507828235Z, scheduled_time=2022-03-28T19:58:06.507828235Z}, vars={}, execution_time=2022-03-28T19:58:06.507945975Z}"
        }

```

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 28, 2022, 8:37pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/18 "2022-03-28T20:37:23Z")

</div>

[This](https://www.elastic.co/guide/en/x-pack/current/input-http.html#input-http-auth-basic-example) (second code block under that heading) is the best I could find in terms of documentation but sending in the API key as a parameter didn't work either, same errors as above.

```auto
"params" :{
          "Authorization": "ApiKey [api key value here]"
        }

```

---

<div class="post-metadata">

**Author:** ![Daniel\_Battaglia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_battaglia/32/49649_2.png) [@Daniel\_Battaglia](https://discuss.elastic.co/u/Daniel_Battaglia)\
**Post date:** [March 28, 2022, 9:23pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/19 "2022-03-28T21:23:24Z")

</div>

Looking at the docs you linked, it seems "auth" only works for type "basic". I do see "request.headers" is an option, so perhaps:

```auto
"request": {
  "headers": {
    "Authorization": "ApiKey [api key value here]"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Apprentice](https://avatars.discourse-cdn.com/v4/letter/a/3bc359/32.png) [@Apprentice](https://discuss.elastic.co/u/Apprentice)\
**Post date:** [March 29, 2022, 1:34pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/20 "2022-03-29T13:34:27Z")

</div>

Tried that as well, still getting:

```auto
"_status_code": 401,
        "errors": [
          {
            "code": "root.unauthenticated",
            "message": "The supplied authentication is invalid"
          }

```

Which is odd because both Postman and Curl are able to use this exact header to get information successfully

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2022, 1:35pm UTC](https://discuss.elastic.co/t/how-to-create-an-automated-solution-for-instance-migration-in-the-case-of-allocator-faliures/300369/21 "2022-04-12T13:35:01Z")

</div>

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.