# Air Gap System: Internal error occurred: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: Service Unavailable **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [June 19, 2019, 7:19am UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396 "2019-06-19T07:19:43Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [June 19, 2019, 7:19am UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/1 "2019-06-19T07:19:43Z")
Hi, I am following the quick start guide: [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html) When I want to deploy the single node elasticsearch I get following error: ``` Internal error occurred: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: Service Unavailable ``` I assume that the problem is that my cluster is behind a proxy server. It can access a private docker registry and can communicate to internet via proxy. https\_proxy is set in docker config and in environment. But later on production there will be NO proxy available. Should ECK work in air-gapped-systems? Thanks, Andreas ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [June 24, 2019, 8:25pm UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/2 "2019-06-24T20:25:38Z")
I don't think the error you are seeing is related to your air-gapped environment. We support that via the image attribute. See [Discussion of custom images](https://discuss.elastic.co/t/using-a-custom-docker-image-for-elastic-cloud-on-kubernetes/184241/7) The error you are seeing is caused by the webhook service that we deploy not being available. We have seen the same error in our e2e test suite and are looking into that. If you have any logs to share that could be helpful in narrowing down what is causing this issue. The operator logs can be retrieved with ```auto kubectl -n elastic-system logs -f statefulset.apps/elastic-operator ``` ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [July 10, 2019, 1:19pm UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/3 "2019-07-10T13:19:05Z")
Hi, my logs are looking like this: > **[View Paste 0oEtb](https://paste.ee/p/0oEtb#s%3D0)** > > Paste.ee - View Paste 0oEtb When I run the command for creating the elasticsearch container, no new log lines are created. Just getting this output: ``` cat < apiVersion: elasticsearch.k8s.elastic.co/v1alpha1 > kind: Elasticsearch > metadata: > name: quickstart > spec: > version: 7.1.0 > nodes: > - nodeCount: 1 > config: > node.master: true > node.data: true > node.ingest: true > EOF Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: Service Unavailable ``` Running on kubernetes 1.14.2 in **bare metal** environment. Internet access is available via proxy only. Docker is currently in dev environment able to access internet via proxy, but will not be able to do so in production later. ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [July 10, 2019, 1:22pm UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/4 "2019-07-10T13:22:16Z")
> [@asp](#): > > [https://elastic-webhook-service.elastic-system.svc:443](https://elastic-webhook-service.elastic-system.svc:443) As I see it want to connect to external webhook service. Do I need to install my own? ---
**Author:** ![kkensy](https://avatars.discourse-cdn.com/v4/letter/k/edb3f5/32.png) [@kkensy](https://discuss.elastic.co/u/kkensy)\ **Post date:** [July 12, 2019, 10:42am UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/5 "2019-07-12T10:42:13Z")
This helped me: > ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [July 24, 2019, 7:57am UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/6 "2019-07-24T07:57:20Z")
> As I see it want to connect to external webhook service. Do I need to install my own? No. This webhook service is auto-created by the operator at startup and is used to validate the Elasticsearch specs you submit to the API server. You have to make sure that your k8s control-plane can reach the webhook service on port 443. See here for details [Timeout when validating admission webhook unreachable · Issue #896 · elastic/cloud-on-k8s · GitHub](https://github.com/elastic/cloud-on-k8s/issues/896#issuecomment-507224945) Alternatively you can remove the validation webhook as a workaround: ```auto kubectl delete validatingwebhookconfigurations validating-webhook-configuration ``` The disadvantage of that workaround is that you lose any immediate feedback on specification errors in your Elasticsearch resources. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 7:12am UTC](https://discuss.elastic.co/t/air-gap-system-internal-error-occurred-failed-calling-webhook-validation-elasticsearch-elastic-co-post-https-elastic-webhook-service-elastic-system-svc-443-validate-elasticsearches-timeout-30s-service-unavailable/186396/7 "2022-11-04T07:12:54Z")