ConnectionError: socket hang up

y34rz3r0 · January 13, 2024, 3:13pm

I successfully run this docker compose locally on macOS and can work with ELK, but when running the same compose I encounter the following errors:

[2024-01-13T14:26:44.001+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: connect ECONNREFUSED 172.18.0.3:9200

And the consequence of this is most likely this error:

{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}

I'm suggesting that the problem might be in IPTABLES, but since I'm new to this technology, I can't figure out what exactly I need to do.

But I see that no packets have been dropped in the DOCKER-ISOLATION-STAGE-2 chain, which makes me confused in my opinion about IPTABLES.

> Chain DOCKER-ISOLATION-STAGE-2 (2 references)
> num   pkts bytes target     prot opt in     out     source               destination         
> 1        0     0 DROP       all  --  *      br-ba4e335d8552  0.0.0.0/0            0.0.0.0/0           
> 2        0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
> 3      111 22701 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Please tell me how can I fix this to run this compose successfully on ubuntu?

Here is the full Kibana log obtained with grep ERROR:

[2024-01-13T14:26:43.966+00:00][ERROR][plugins.securitySolution.endpoint:user-artifact-packager:1.0.0] EndpointError: Error scheduling task, received socket hang up - Local: 172.18.0.4:38172, Remote: unknown:unknown
[2024-01-13T14:26:43.968+00:00][ERROR][plugins.taskManager] Task cases-telemetry-task "cases-telemetry-task" failed: ConnectionError: socket hang up - Local: 172.18.0.4:38102, Remote: 172.18.0.3:9200
[2024-01-13T14:26:43.970+00:00][ERROR][plugins.taskManager] Task Fleet-Usage-Sender "Fleet-Usage-Sender-1.1.0" failed: ConnectionError: read ECONNRESET - Local: unknown:unknown, Remote: unknown:unknown
[2024-01-13T14:26:44.001+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: connect ECONNREFUSED 172.18.0.3:9200
[2024-01-13T14:26:44.177+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.186+00:00][ERROR][plugins.fleet] Failed to fetch latest version of synthetics from registry: '403 Forbidden' error response from package registry at https://epr.elastic.co/search?package=synthetics&prerelease=true&kibana.version=8.9.1
[2024-01-13T14:26:44.247+00:00][ERROR][plugins.fleet] uninstalling synthetics-1.0.1 after error installing: [ConnectionError: getaddrinfo EAI_AGAIN elasticsearch]
[2024-01-13T14:26:44.251+00:00][ERROR][plugins.fleet] failed to update package status to: install_failed  ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.252+00:00][ERROR][plugins.fleet] failed to uninstall or rollback package after installation error Error: synthetics is not installed
[2024-01-13T14:26:44.252+00:00][ERROR][plugins.synthetics] ConnectionError: Error installing synthetics 1.0.1: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.416+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.648+00:00][ERROR][plugins.taskManager] Task security:endpoint-diagnostics "security:endpoint-diagnostics:1.0.0" failed: ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.649+00:00][ERROR][plugins.taskManager] Task security:endpoint-meta-telemetry "security:endpoint-meta-telemetry:1.0.0" failed: ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.800+00:00][ERROR][plugins.security.authorization] Error registering Kibana Privileges with Elasticsearch for kibana-.kibana: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:44.802+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:47.798+00:00][ERROR][plugins.alerting] Error installing index template .alerts-security.alerts-default-index-template - getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:47.798+00:00][ERROR][plugins.alerting] ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:47.799+00:00][ERROR][plugins.alerting] Error initializing context security - Failure during installation. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:51.210+00:00][ERROR][plugins.alerting] Error installing index template .alerts-observability.logs.alerts-default-index-template - getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:51.211+00:00][ERROR][plugins.alerting] ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:51.212+00:00][ERROR][plugins.alerting] Error initializing context observability.logs - Failure during installation. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:52.223+00:00][ERROR][plugins.alerting] Error creating concrete write index - getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:52.223+00:00][ERROR][plugins.alerting] ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:52.224+00:00][ERROR][plugins.alerting] Error initializing context observability.uptime - Failure during installation. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:52.836+00:00][ERROR][plugins.alerting] Error creating concrete write index - getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:52.836+00:00][ERROR][plugins.alerting] ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:52.837+00:00][ERROR][plugins.alerting] Error initializing context observability.slo - Failure during installation. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:54.416+00:00][ERROR][plugins.alerting] Error installing index template .alerts-observability.metrics.alerts-default-index-template - getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:54.417+00:00][ERROR][plugins.alerting] ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:54.418+00:00][ERROR][plugins.alerting] Error initializing context observability.metrics - Failure during installation. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:55.047+00:00][ERROR][plugins.alerting] Error creating concrete write index - getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:55.047+00:00][ERROR][plugins.alerting] ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:26:55.048+00:00][ERROR][plugins.alerting] Error initializing context observability.apm - Failure during installation. getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:27:13.951+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:27:24.052+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:27:26.646+00:00][ERROR][plugins.eventLog] error initializing elasticsearch resources: error checking existance of ilm policy: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:27:26.647+00:00][ERROR][plugins.eventLog] initialization failed, events will not be indexed
[2024-01-13T14:27:34.135+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:27:44.208+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:27:54.275+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:28:04.366+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:28:14.441+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:28:24.537+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:28:34.650+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:28:44.744+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:28:54.816+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:29:04.888+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:29:14.960+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:29:25.055+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:29:35.143+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:29:45.228+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:29:55.308+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:30:05.382+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:30:15.456+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:30:25.539+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:30:35.612+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:30:45.700+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:30:55.769+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:31:05.850+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:31:15.921+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:31:26.010+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:31:36.103+00:00][ERROR][plugins.security.authentication] License is not available, authentication is not possible.
[2024-01-13T14:31:38.140+00:00][ERROR][savedobjects-service.repository.point-in-time-finder] Failed to open PIT for types [application_usage_totals]
ConnectionError: getaddrinfo EAI_AGAIN elasticsearch
[2024-01-13T14:31:38.143+00:00][ERROR][savedobjects-service.repository.point-in-time-finder] Failed to open PIT for types [application_usage_daily]
    at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:528:31)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at KibanaTransport.request (/usr/share/kibana/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
    at ClientTraced.GetApi [as get] (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/get.js:36:12) {
  meta: {
    body: undefined,
    statusCode: 0,
    headers: {},
    meta: {
      context: null,
      request: [Object],
      name: 'elasticsearch-js',
      connection: [Object],
      attempts: 0,
      aborted: false
    },
    warnings: [Getter]
  },
  isBoom: true,
  isServer: true,
  data: null,
  output: {
    statusCode: 503,
    payload: {
      statusCode: 503,
      error: 'Service Unavailable',
      message: 'getaddrinfo EAI_AGAIN elasticsearch'
    },
    headers: {}
  },
  [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/esUnavailable'
}

tsullivan · January 19, 2024, 11:34pm

Your docker-compose looks very similar to one in our documentation. I would suggest starting with that one exactly as-is, and then gradually customize it as needed.

See: Install Elasticsearch with Docker | Elasticsearch Guide [8.12] | Elastic
And: elasticsearch/docs/reference/setup/install/docker/docker-compose.yml at 8.12 · elastic/elasticsearch · GitHub

y34rz3r0 · January 27, 2024, 12:32pm

Yes, this is the official compose from Elastic, adapted to my needs, that’s why I asked for help here. Unfortunately, this does not answer my question.

tsullivan · January 29, 2024, 3:57pm

Are you able to run all 3 nodes and get a Kibana connection using the "unadapted" docker compose configuration?

y34rz3r0 · February 2, 2024, 2:37pm

I took the original compose and env provided by Elastic:
compose: https://raw.githubusercontent.com/elastic/elasticsearch/8.12/docs/reference/setup/install/docker/docker-compose.yml
env:
https://raw.githubusercontent.com/elastic/elasticsearch/8.12/docs/reference/setup/install/docker/.env

I specified 8.9.1 as the version.

Here's the same error:

{"statusCode":503,"error":"Service Unavailable","message":"License is not available."}

For convenience, I have included the full log of the docker-compose.yml up command in my repository:

github.com

y34r-z3r0/elastic/blob/main/compose.log

Creating network "elastic_default" with the default driver
Creating volume "elastic_certs" with local driver
Creating volume "elastic_esdata01" with local driver
Creating volume "elastic_esdata02" with local driver
Creating volume "elastic_esdata03" with local driver
Creating volume "elastic_kibanadata" with local driver
Creating elastic_setup_1 ... done
Creating elastic_es01_1  ... done
Creating elastic_es02_1  ... done
Creating elastic_es03_1  ... done
Creating elastic_kibana_1 ... done
Attaching to elastic_setup_1, elastic_es01_1, elastic_es02_1, elastic_es03_1, elastic_kibana_1
es01_1    | Created elasticsearch keystore in /usr/share/elasticsearch/config/elasticsearch.keystore
es02_1    | {"@timestamp":"2024-02-02T14:23:42.267Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.util.VectorUtilPanamaProvider","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
es01_1    | {"@timestamp":"2024-02-02T14:23:47.692Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.util.VectorUtilPanamaProvider","elasticsearch.node.name":"es01","elasticsearch.cluster.name":"docker-cluster"}
es02_1    | {"@timestamp":"2024-02-02T14:23:43.114Z", "log.level": "INFO", "message":"version[8.9.1], pid[67], build[docker/a813d015ef1826148d9d389bd1c0d781c6e349f0/2023-08-10T05:02:32.517455352Z], OS[Linux/5.15.0-91-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.2/20.0.2+9-78]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
es02_1    | {"@timestamp":"2024-02-02T14:23:43.115Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es02","elasticsearch.cluster.name":"docker-cluster"}
es03_1    | {"@timestamp":"2024-02-02T14:23:42.621Z", "log.level": "INFO", "message":"Java vector incubator API enabled; uses preferredBitSize=256", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.apache.lucene.util.VectorUtilPanamaProvider","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"docker-cluster"}
es03_1    | {"@timestamp":"2024-02-02T14:23:43.424Z", "log.level": "INFO", "message":"version[8.9.1], pid[66], build[docker/a813d015ef1826148d9d389bd1c0d781c6e349f0/2023-08-10T05:02:32.517455352Z], OS[Linux/5.15.0-91-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/20.0.2/20.0.2+9-78]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"docker-cluster"}
es03_1    | {"@timestamp":"2024-02-02T14:23:43.426Z", "log.level": "INFO", "message":"JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.node.Node","elasticsearch.node.name":"es03","elasticsearch.cluster.name":"docker-cluster"}

This file has been truncated. show original

system · March 1, 2024, 2:38pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.