Air Gap System: Internal error occurred: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: Service Unavailable

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi,

I am following the quick start guide: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html

When I want to deploy the single node elasticsearch I get following error:

Internal error occurred: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: Service Unavailable

I assume that the problem is that my cluster is behind a proxy server. It can access a private docker registry and can communicate to internet via proxy. https_proxy is set in docker config and in environment. But later on production there will be NO proxy available.

Should ECK work in air-gapped-systems?

Thanks,
Andreas

2

I don't think the error you are seeing is related to your air-gapped environment. We support that via the image attribute. See Discussion of custom images

The error you are seeing is caused by the webhook service that we deploy not being available. We have seen the same error in our e2e test suite and are looking into that.

If you have any logs to share that could be helpful in narrowing down what is causing this issue.

The operator logs can be retrieved with

kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
3

Hi, my logs are looking like this:

When I run the command for creating the elasticsearch container, no new log lines are created. Just getting this output:

 cat <<EOF | kubectl apply -f -
> apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
> kind: Elasticsearch
> metadata:
>   name: quickstart
> spec:
>   version: 7.1.0
>   nodes:
>   - nodeCount: 1
>     config:
>       node.master: true
>       node.data: true
>       node.ingest: true
> EOF
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: Service Unavailable

Running on kubernetes 1.14.2 in bare metal environment. Internet access is available via proxy only. Docker is currently in dev environment able to access internet via proxy, but will not be able to do so in production later.

4

As I see it want to connect to external webhook service. Do I need to install my own?

5

This helped me:

6

As I see it want to connect to external webhook service. Do I need to install my own?

No. This webhook service is auto-created by the operator at startup and is used to validate the Elasticsearch specs you submit to the API server.

You have to make sure that your k8s control-plane can reach the webhook service on port 443. See here for details Timeout when validating admission webhook unreachable · Issue #896 · elastic/cloud-on-k8s · GitHub

Alternatively you can remove the validation webhook as a workaround:

kubectl delete validatingwebhookconfigurations validating-webhook-configuration

The disadvantage of that workaround is that you lose any immediate feedback on specification errors in your Elasticsearch resources.