I'm using docker compose to put together some containers running code that my team has built, plus rabbitmq, and also an elk stack to monitor the rabbitmq logs and queues. I'm attempting to follow the configuration in:
Something's not working with the containers talking to each other. When I log into the elasticsearch container to set the main password so I can set up accounts, I get this:
elasticsearch@7ba72de0917c:~$ /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
Unexpected response code [405] from calling GET http://172.18.0.7:9200/_security/_authenticate?pretty
It doesn't look like the X-Pack security feature is enabled on this Elasticsearch node.
Please check if you have enabled X-Pack security in your elasticsearch.yml configuration file.
ERROR: X-Pack Security is disabled by configuration.
with the security turned off. When I turn it on and do the same thing, I immediately get errors starting with
19:15:11.510 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [172.18.0.7]; the server provided a certificate with subject name [CN=es01], fingerprint
So why in both cases is it trying to contact 172.18.* network? I'm pretty sure that's not on my network. Why is it contacting an external server? How do I change this?
I cannot find anywhere a set of working set of docker-compose.yml files plus elasticsearch.yml files that are known good to start with. The elastic documentation certainly doesn't have them. Without known good examples I'm working without anything valid to start with and it's very frustrating.
As mentioned by @rugenl this range is a private network, not an external server, the network 172.18.0.0/16 is one of the default Docker networks.
Use docker network ls to list the networks your docker is using and then docker network inspect <network-name> to get more information them, including the IP address range being used.
Yes, indeed. I know the class-A IP address by heart (10...) and the class-C (192.168...) but I guess not the class B (172.16-31...). Thank you for pointing that out specifically.
So this is just the standardin, default, internal network (sensibly) set up by docker compose. Great. Makes perfect sense. My fundamental problem remains, however, that I can't figure out who those two things don't talk to each other. I will post my docker-compose file in response to the other comment.
Thank you for your help so far, everyone. Ok, I understand that the IP address is just the default docker-compose internal network. I'm going to paste in snippets of my docker-compose.yml file in case someone can point to the bit that's making it not work. (I will paste the whole thing if that helps, but I don't see a way to attach a file, and it's 329 lines, which probably is a bit big for a comment.)
By the way, I've run this a bunch of times. Sometimes I try to turn on all the security as it generally recommends, and sometimes I try to turn it all off. I make the config files for the applications (kibana and elasticsearch) match the docker-compose file in that respect. Both on and off, it comes up with the very similar errors, which tends to tell me that I have something misconfigured in the network side. But I'm leaving the network section blank, so I don't know what the problem is.
Here's the initializer:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
volumes:
- certs:/usr/share/elasticsearch/config/certs
user: "0"
command: >
bash -c '
if [ x${ELASTIC_PASSWORD} == x ]; then
echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
exit 1;
elif [ x${KIBANA_PASSWORD} == x ]; then
echo "Set the KIBANA_PASSWORD environment variable in the .env file";
exit 1;
fi;
if [ ! -f config/certs/ca.zip ]; then
echo "Creating CA";
bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
unzip config/certs/ca.zip -d config/certs;
fi;
if [ ! -f config/certs/certs.zip ]; then
echo "Creating certs";
echo -ne \
"instances:\n"\
" - name: es01\n"\
" dns:\n"\
" - es01\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
" - name: kibana\n"\
" dns:\n"\
" - kibana\n"\
" - localhost\n"\
" ip:\n"\
" - 127.0.0.1\n"\
> config/certs/instances.yml;
bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
unzip config/certs/certs.zip -d config/certs;
fi;
echo "Setting file permissions"
chown -R root:root config/certs;
find . -type d -exec chmod 750 \{\} \;;
find . -type f -exec chmod 640 \{\} \;;
echo "Waiting for Elasticsearch availability";
until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
'
healthcheck:
test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
interval: 1s
timeout: 5s
retries: 120
Thank anyone for your help. What I'm mostly looking for is what would be obvious for someone who uses these tools but I don't see because I don't know what the minimum required configuration is.
Ok, since the comments embed files in a reasonable way, here's my entire docker-compose.yml file.
traefik and rabbitmq are packaged services that we use. cdrhook, golden_muscat, downloader, uploader are our code and are known to work. Everything else is my attempt to create a working elk configuration.
cluster.name: "elasticsearch"
server.name: es01
#network.host: localhost
network.host: 0.0.0.0
# getting ssl to work
xpack.security.enabled: false
xpack.security.autoconfiguration.enabled: true
The configuration you see here is with the security flags turned off. On is with all of those simultaneously turned on. Both give error messages; the error messages are in the original post.
The software refuses to start if the Xpack security is turned on. But if it's on with all the features are turned on, it kept getting connect errors every time it tried to connect.
However, if I set Xpack security to ON, but SSL checking within Xpack to be off (here's the Elasticsearch snippet from docker-compose.yml; the elasticsearch.yml has to match)
it now seems to come up with this when running the setup passwords script:
elasticsearch@75ba47422194:~$ /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
******************************************************************************
Note: The 'elasticsearch-setup-passwords' tool has been deprecated. This command will be removed in a future release.
******************************************************************************
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]
That's a warning not an error. We can help you resolve the warning, but it's not the direct cause of whatever problem you have (unless the only problem you have is that there are too many warnings in the logs).
Let's take a step back.
Is your compose file working?
If not, how do you know it's not working (not, "what do you think is the cause" but how did you determine that it's not working - what did you try to do that failed?)
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.