Kibana gets connection refused from elasticsearch even if xpack.security.enabled: false is set

Elastic Search
1

docker-compose.yml:

elasticsearch:
  container_name: "elasticsearch_instance"
  image: elasticsearch:8.3.2
  environment:
    - discovery.type=single-node
    - ES_JAVA_OPTS=-Xms1g -Xmx1g
    - xpack.security.enabled=false
  volumes:
    - /home/username/temp/elastic_data:/usr/share/elasticsearch/data
    - /home/username/configs/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  ports:
    - 9200:9200
kibana:
  image: kibana:8.3.2
  ports:
    - target: 5601
      published: 5601
  environment:
    - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
  depends_on:
    - elasticsearch

elasticsearch.yml:

discovery.type: single-node
xpack.security.enabled: false
ingest.geoip.downloader.enabled: false

kibana.yml was unchanged (should be the autogenerated one).

Issue:
Starting the docker via "docker compose up", I get this message in the logs:

kibana-1                          | [2024-07-12T12:45:28.079+00:00][INFO ][plugins-service] Plugin "cloudSecurityPosture" is disabled.
kibana-1                          | [2024-07-12T12:45:28.159+00:00][INFO ][http.server.Preboot] http server running at http://0.0.0.0:5601
kibana-1                          | [2024-07-12T12:45:28.189+00:00][INFO ][plugins-system.preboot] Setting up [1] plugins: [interactiveSetup]
kibana-1                          | [2024-07-12T12:45:28.191+00:00][INFO ][preboot] "interactiveSetup" plugin is holding setup: Validating Elasticsearch connection configuration…
kibana-1                          | [2024-07-12T12:45:28.213+00:00][INFO ][root] Holding setup until preboot stage is completed.
kibana-1                          |
kibana-1                          |
kibana-1                          | i Kibana has not been configured.
kibana-1                          |
kibana-1                          | Go to http://0.0.0.0:5601/?code=873462 to get started.
kibana-1                          |
kibana-1                          |

When I go to localhost:5601 on my machine where the cluster is running via Windows Subsystem for Linux, it asks for an enrollment tocken. I cannot create an enrollment tocken since xpack.security.enabled: false is set.

If I go into the kibana docker and try to ping the elasticsearch instance it is reachable from the kibana container:

root@e68cdb6bac9f:/usr/share/kibana# ping elasticsearch
PING elasticsearch (172.19.0.4) 56(84) bytes of data.
64 bytes from elasticsearch_instance.agentconfigs_default (172.19.0.4): icmp_seq=1 ttl=64 time=0.073 ms
64 bytes from elasticsearch_instance.agentconfigs_default (172.19.0.4): icmp_seq=2 ttl=64 time=0.184 ms
64 bytes from elasticsearch_instance.agentconfigs_default (172.19.0.4): icmp_seq=3 ttl=64 time=0.082 ms
64 bytes from elasticsearch_instance.agentconfigs_default (172.19.0.4): icmp_seq=4 ttl=64 time=0.124 ms

However if I try to curl it, I get a connection refused:

root@e68cdb6bac9f:/usr/share/kibana# curl elasticsearch:9200
curl: (7) Failed to connect to elasticsearch port 9200: Connection refused

Same if I try to use the IP of the elasticsearch instance instead:

root@e68cdb6bac9f:/usr/share/kibana# ping 172.19.0.4
PING 172.19.0.4 (172.19.0.4) 56(84) bytes of data.
64 bytes from 172.19.0.4: icmp_seq=1 ttl=64 time=0.096 ms
64 bytes from 172.19.0.4: icmp_seq=2 ttl=64 time=0.172 ms
^C
--- 172.19.0.4 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1015ms
rtt min/avg/max/mdev = 0.096/0.134/0.172/0.038 ms
root@e68cdb6bac9f:/usr/share/kibana# curl 172.19.0.4:9200
curl: (7) Failed to connect to 172.19.0.4 port 9200: Connection refused

If I go into the elasticsearch container and try to curl localhost this is the result:

elasticsearch@1d7e418d7cde:~$ curl localhost:9200
{
  "name" : "1d7e418d7cde",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "xeJ4eBMgQeiyx5qNcs6BEw",
  "version" : {
    "number" : "8.3.2",
    "build_type" : "docker",
    "build_hash" : "8b0b1f23fbebecc3c88e4464319dea8989f374fd",
    "build_date" : "2022-07-06T15:15:15.901688194Z",
    "build_snapshot" : false,
    "lucene_version" : "9.2.0",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}

It seems the elasticsearch is reachable, and kibana asks for an enrollment token even if security is turned off. What did I miss?

2

Any help, or ideas how to debug/investigate are highly appreciated. I am stuck and don't understand why I get "Connection Refused" even if all security should be deactivated...

3

Hey @TheFox

You might also need to explicitly disable security in Kibana by adding the following:

environment:
      - XPACK_SECURITY_ENABLED=false

If you can't connect to Elasticsearch from within the Kibana container via curl, it suggests a Docker networking issue, and it is not necessarily related to the Elasticsearch configuration.

4

Hi,

thanks for the answer.

I added the suggested option in the kibana section of my docker compose file:

  kibana:
    image: kibana:8.3.2
    ports:
      - target: 5601
        published: 5601
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - XPACK_SECURITY_ENABLED=false
    depends_on:
      - elasticsearch

I did a docker network inspect to check if the containers are on the same network and I can see the elasticsearch_instance and the kibana instance on the same network.

user@machine:~$ docker network inspect bb1623004dd4
[
    {
        "Name": "agentconfigs_default",
        "Id": "bb1623004dd40a18ad797347477acacd26954841cefcf9210be391e75f5c767d",
        "Created": "2024-06-28T14:27:28.118589498+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.19.0.0/16",
                    "Gateway": "172.19.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "1d7e418d7cde023a080c0b287cfea3a1865495675434310fabe3bc22faa362c8": {
                "Name": "elasticsearch_instance",
                "EndpointID": "b5966c42370018cd9329d1d903aa8d38d28a913f98ada7a6035dc1c2701e3775",
                "MacAddress": "02:42:ac:13:00:02",
                "IPv4Address": "172.19.0.2/16",
                "IPv6Address": ""
            },
            "57d2fec4f78f6b45a273854ea3864d552d14c388a2abe38a19d9c28449047670": {
                "Name": "agentconfigs-kibana-1",
                "EndpointID": "36808fc2ea594b7ec609f724daf6e91c564bee8e43f2541be616dfe57f9ac851",
                "MacAddress": "02:42:ac:13:00:05",
                "IPv4Address": "172.19.0.5/16",
                "IPv6Address": ""
            },
},
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "agentconfigs",
            "com.docker.compose.version": "2.27.0"
        }
    }
]

Is there a better way to verify the network behavior / connection?

If it is a network issue, I don't really understand it, because the containers can ping each other, so they can see each other.

Is there a better way to debug on the elasticsearch side which connection requests there were, and probably also getting the reason why the connection was refused?

5

If I switch to the versions from this example docker-compose.yml (and ofc change the configuration accordingly), it works.

https://github.com/elastic/ecs-dotnet/blob/main/examples/console-with-extensions-logging/docker-compose.yml

Tbh, I don't think it is a docker network issue. It would be great if someone could update the ecs-dotnet examples to the newest or at least an 8.x version of elasticsearch.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.