How to create an automated solution for Instance Migration in the case of Allocator Faliures?

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

I've found this for the manual method: Move nodes or instances from allocators | Elastic Cloud Enterprise Reference [3.1] | Elastic

I looked at the APIs here but couldn't find anything relevant to my situation.

I'm new to Elastic and am not sure were to start looking so would appreciate any assistance in pointing me to the right direction.

2

I did manage to find some docs that could help, I could use watcher to create an alert which could then trigger the migration

3

I'm currently working on testing the sensing part of the watcher using the following query but I'm having trouble accessing the API using my API key (not sure how to setup the key, I've only seen examples for user/password setup).

{
  "trigger": {
    "schedule": {
      "interval": "2m"
    }
  },
  "input": {
    "http" : {
      "request" : {
        "scheme": "http",
        "host" : "$COORDINATOR_HOST",
        "port" : 12443,
        "path" : "/api/v1/platform/infrastructure/allocators",
        "method": "get",
        "auth":{
                  "Authorization": "ApiKey $ECE_API_KEY"
        }
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "{{ctx}}"
      }
    }
  }
}

Because without the authorization I get no response from the server:

"error": {
"root_cause": [
{
"type": "no_http_response_exception",
"reason": "$COORDINATOR_HOST:12443 failed to respond"
}

But I know that it can send a response because I've tested it on a terminal and got the expected response:

curl -k -X GET -H "Authorization: ApiKey $ECE_API_KEY" https://$COORDINATOR_HOST:12443/api/v1/platform/infrastructure/allocators
4

Based on that error it looks more likely that it's not actually reaching the correct URL than anything to do with the API key (which I would expect a 401 response, not no_http_response_exception). If you put the hardcoded value for the ECE host instead of $COORDINATOR_HOST does it work?

5

I have hardcoded in the ECE host value, I'm not actually pointing to any variable.

I'm confused because I use the same information for the terminal command as I mentioned previously and it worked fine.

6

Hmm I just noticed, you are using the HTTPS port 12443 but a scheme of http. Can you change the scheme to https or the port to 12400?

7

changed the scheme to https, but now I'm getting the following error:

"error": {
"root_cause": [
{
"type": "s_s_l_handshake_exception",
"reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
}

I should mention that I am trying to use a watcher to pull info from the ECE API, will that cause any issues?

8

That looks like the JVM doesn't know about your SSL certificate. I think you might need to use one from a known public CA. Maybe try for now using http and port 12400 while testing out the overall workflow? I don't see any reason why this should have issues hitting the ECE API if everything is configured correctly.

9

Using http and port 12400 gives me the following error:

"type": "http",
"status": "failure",
"error": {
"root_cause": [
{
"type": "http_host_connect_exception",
"reason": "Connect to [hostname] failed: Connection refused"
}
],
"type": "http_host_connect_exception",
"reason": "Connect to [hostname] failed: Connection refused",
"caused_by": {
"type": "connect_exception",
"reason": "Connection refused"
}
},

I noticed the curl command specified in the docs uses the -k option, "this option makes curl skip the verification step and proceed without checking". Is there any similar option I can use in the watcher?

10

This is a general networking error, I guess the URL isn't valid (maybe you are using a load balancer in front of ECE and not exposing the regular http port?). Are you able to use curl to http://$COORDINATOR_HOST:12400"? If that works I'd expect the watcher to work as well (no need for -k here since there is no SSL certificates).

I'm not sure if watcher allows you to bypass SSL certificate warnings but I'd imagine it doesn't as this is insecure, you should generally either use regular HTTP or else HTTPS with a valid certificate.

11

Thats why I find it so odd, I'm able to use Curl to get the information but not the watcher. The exact curl command I used is:

curl -k -X GET -H "Authorization: ApiKey [key value here]" https://[host url here]:12443/api/v1/platform/infrastructure/allocators

And the watcher query is as follows:

{
  "trigger": {
    "schedule": {
      "interval": "2m"
    }
  },
  "input": {
    "http" : {
      "request" : {
        "scheme": "http",
        "host" : "[host url here]",
        "port" : 12400,
        "path" : "/api/v1/platform/infrastructure/allocators",
        "method": "get",
        "auth":{
          "Authorization": "ApiKey [key value here]"
        }
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "{{ctx}}"
      }
    }
  }
}

This gives me the following error:

      "status": "failure",
      "error": {
        "root_cause": [
          {
            "type": "http_host_connect_exception",
            "reason": "Connect to [host url here]:12400 failed: Connection refused"
          }
        ],
        "type": "http_host_connect_exception",
        "reason": "Connect to  [host url here]:12400 failed: Connection refused",
        "caused_by": {
          "type": "connect_exception",
          "reason": "Connection refused"
        }
      },

When I change the scheme to https and port to 12443, I get the certs error:

"type": "http",
      "status": "failure",
      "error": {
        "root_cause": [
          {
            "type": "s_s_l_handshake_exception",
            "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
          }
        ],
        "type": "s_s_l_handshake_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "validator_exception",
          "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
          "caused_by": {
            "type": "sun_cert_path_builder_exception",
            "reason": "unable to find valid certification path to requested target"
          }
        }
      },
12

I'm simulating the watcher in Cloud UI > Stack Management > Watcher > Create, so it's not even like I'm trying to make a request from outside

13

This makes sense though, the -k option is telling curl you don't care about the SSL certificate not being valid. I would test anything in curl that you want to run in watcher without using -k.

14

So watcher doesn't have anything similar to ignore verification?

15

Not that I'm aware of (I'm on the ECE team though so I can't really speak as a Watcher expert).

17

I've found xpack.http.ssl.verification_mode parameter to help me ignore the certs issue to allow testing for now and I think I am able to reach the ECE API now.

However it seems that the way I am sending the ECE api key via the get request is incorrect and I'm not sure of the correct format as all examples in the doc only show me a username/password example

I am getting the following error:

errors=[{code=root.unauthenticated, message=The supplied authentication is invalid}]}

Full Response to the action:

"logged_text": "{metadata={name=allocator_monitoring, xpack={type=json}}, watch_id=_inlined_, payload={_headers={x-request-id=[*alpha-numeric stuff here*], date=[Mon, 28 Mar 2022 19:58:06 GMT], server=[*alpha-numeric stuff here*], transfer-encoding=[chunked], vary=[Accept-Encoding], x-frame-options=[SAMEORIGIN], www-authenticate=[Bearer realm=\"found-adminconsole\"], x-download-options=[noopen], strict-transport-security=[max-age=3600; includeSubDomains], set-cookie=[ec-session=; Max-Age=0; Path=/; Secure; HttpOnly], content-security-policy=[default-src 'none';script-src 'self';worker-src 'self' blob:;connect-src 'self' https://telemetry.elastic.co;img-src 'self' data:;style-src 'self' 'unsafe-inline';manifest-src 'self';font-src 'self';frame-src 'self'], x-content-type-options=[nosniff], x-xss-protection=[1;mode=block], content-type=[application/json], connection=[Keep-Alive], x-ui-tag=[indeterminate], x-cloud-error-codes=[root.unauthenticated], cache-control=[max-age=0, no-cache, no-store, must-revalidate]}, _status_code=401, errors=[{code=root.unauthenticated, message=The supplied authentication is invalid}]}, id=_inlined__*alpha-numeric stuff here*-2022-03-28T19:58:06.507945975Z, trigger={triggered_time=2022-03-28T19:58:06.507828235Z, scheduled_time=2022-03-28T19:58:06.507828235Z}, vars={}, execution_time=2022-03-28T19:58:06.507945975Z}"
        }
18

This (second code block under that heading) is the best I could find in terms of documentation but sending in the API key as a parameter didn't work either, same errors as above.

"params" :{
          "Authorization": "ApiKey [api key value here]"
        }
19

Looking at the docs you linked, it seems "auth" only works for type "basic". I do see "request.headers" is an option, so perhaps:

"request": {
  "headers": {
    "Authorization": "ApiKey [api key value here]"
  }
}
20

Tried that as well, still getting:

"_status_code": 401,
        "errors": [
          {
            "code": "root.unauthenticated",
            "message": "The supplied authentication is invalid"
          }

Which is odd because both Postman and Curl are able to use this exact header to get information successfully

21

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.