Hi,
I have setup an elaticsearch cluster of 3 nodes.
I'm running the latest version of elasticsearch: 7.3.2.
After setting the xpack.security.enabled to true, the cluster has stopped working.
After running curl -x to check the health of the cluster, it says that it failed to connect to the node and that the connection is refused.
Can you help me with this issue ?
Thank you,
Please share your configuration and the logs from the elasticsearch nodes. Without this information, it's practically impossible for anyone to help you.
1 Like
Hi,
Here is the elasticsearch.yml configuration file:
cluster.name: cluster-elk-prod
node.name: elk-1 #elk-2 and elk-3 on the other nodes
path.data: /var/lib/elastic
path.logs: /var/log/elasticsearch
network.host: 10.0.1.4
discovery.seed_hosts: ["10.0.1.4", "10.0.1.5", "10.0.1.6"]
cluster.initial_master_nodes: ["elk-1"]
xpack.security.enabled: true
Here are the logs of the cluster:
[1] bootstrap checks failed
[1]: Transport SSL must be enabled if security is enabled on a [basic] license. Please set [xpack.security.transport.ssl.enabled] to [true] or disable security by setting [xpack.security.enabled] to [false]
[2019-09-27T14:21:09,806][INFO ][o.e.n.Node ] [elk-1] stopping ...
[2019-09-27T14:21:09,838][INFO ][o.e.n.Node ] [elk-1] stopped
[2019-09-27T14:21:09,839][INFO ][o.e.n.Node ] [elk-1] closing ...
[2019-09-27T14:21:09,855][INFO ][o.e.n.Node ] [elk-1] closed
[2019-09-27T14:21:09,857][INFO ][o.e.x.m.p.NativeController] [elk-1] Native controller process has stopped - no new native processes can be started
Thank you,
Results of checking the health of the cluster:
curl -X GET http://10.0.1.4:9200/_cluster/health?pretty
curl: (7) Failed to connect to 10.0.1.4 port 9200: Connection refused
You need to enable TLS when you use security. See our docs on how to do this : Configure TLS | Elasticsearch Guide [8.11] | Elastic
I have tried enabling TLS, but I don't know how to use the same CA for all the nodes in the cluster.
Should I generate a single CA and a single certificate to be used by all the nodes ? In this case, how can I transfer these files (CA and certificate) to the other nodes ?
Or should I generate a CA and certificate for every node, separately ?
I couldn't find this info in the documentation.
Thank you,
Everything that you ask can be found in the docs I shared with you, please read through it again https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html#node-certificates
Please don't post unformatted code, logs, or configuration as it's very hard to read.
Instead, paste the text and format it with </> icon or pairs of triple backticks (```), and check the preview window to make sure it's properly formatted before posting it. This makes it more likely that your question will receive a useful answer.
It would be great if you could update your post to solve this.
- Do you actually need to run one Kibana instance per docker container ?
- The docker logs should be much more than this one single line. Please share relevant logs from both kibana and elasticsearch.
The more time you put into providing some very basic and necessary pieces of information up front, the quicker you'll get to be assisted towards a resolution !
Hi,
I want to run one Kibana docker per instance container for redundancy purpose.
These are couple of logs of the Kibana docker, associated to the authentication event. I can't paste all of them, due to the maximum characters permitted. I don't see any relevant logs coming from the elasticsearch cluster. (/var/log/elasticsearch/cluster-elk-prod.log)
{"type":"response","@timestamp":"2019-10-01T08:59:05Z","tags":[],"pid":6,"method":"get","statusCode":304,"req":{"url":"/bundles/login.bundle.js","method":"get","headers":{"host":"servers","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/login?next=%2F","if-none-match":"\"e0c275e902beb3a821a96713d4988bae15e8bba3-/bundles/-gzip\""},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/login?next=%2F"},"res":{"statusCode":304,"responseTime":3,"contentLength":9},"message":"GET /bundles/login.bundle.js 304 3ms - 9.0B"}
{"type":"response","@timestamp":"2019-10-01T08:59:06Z","tags":[],"pid":6,"method":"get","statusCode":304,"req":{"url":"/built_assets/dlls/icon.logo_kibana-js.bundle.dll.js","method":"get","headers":{"host":"servers","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/login?next=%2F","if-none-match":"\"7fd622cd3f0956ac15a6ca864b27d30e76123cef-/built_assets/dlls/-gzip\""},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/login?next=%2F"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /built_assets/dlls/icon.logo_kibana-js.bundle.dll.js 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-10-01T08:59:06Z","tags":[],"pid":6,"method":"get","statusCode":304,"req":{"url":"/built_assets/dlls/icon.clock-js.bundle.dll.js","method":"get","headers":{"host":"servers","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0","accept":"*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/login?next=%2F","if-none-match":"\"b01edb462212cd7ba943db47d90dad3cff42e59a-/built_assets/dlls/-gzip\""},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/login?next=%2F"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /built_assets/dlls/icon.clock-js.bundle.dll.js 304 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-10-01T08:59:06Z","tags":[],"pid":6,"method":"get","statusCode":304,"req":{"url":"/ui/images/bg_top_branded.svg","method":"get","headers":{"host":"servers","connection":"close","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0","accept":"image/webp,*/*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate, br","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/built_assets/css/plugins/security/index.light.css","if-modified-since":"Fri, 06 Sep 2019 15:21:47 GMT","if-none-match":"\"db3622756413533cdb3a029b8a6b4e26380bf693-gzip\""},"remoteAddress":"172.17.0.1","userAgent":"172.17.0.1","referer":"https://secops-elk1.westeurope.cloudapp.azure.com/built_assets/css/plugins/security/index.light.css"},"res":{"statusCode":304,"responseTime":2,"contentLength":9},"message":"GET /ui/images/bg_top_branded.svg 304 2ms - 9.0B"}Apologies but I'll have to repeat myself
There's nothing we can do to help you out with the amount of information you share with us.
Could please point out the name of the log files? I ran #docker logs kibana and I have looked into /var/log/elasticsearch/cluster.log.
Start elasticsearch and kibana containers with the configuration that doesn't work. Then run docker ps to get the running instances and then run docker logs <instance_id> --since 10m on all of them and share the output .If it doesn't fit here, you can use a text hosting service like https://gist.github.com or https://pastebin.com/
Please provide all the information you are asked to.
These are all the logs from the containers.
Thank you
Can't do anything unless you share your elasticsearch logs.
Hi,
Thank you for your support.
I solved the problem. It was coming from the nginx load balancer which was using the Round Robin method, without any directives implemented.
I modified the nginx configuration file with the ip_hash directive and now Kibana is running with all 3 dockers up&running.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.