ConnectionError while creating track in Rally with docker image

[akm]

Hi,

I am trying to create a custom track from an existing Elasticsearch cluster using the instructions in the Rally documentation page.

I am running Rally as a Docker Image. I am running elasticsearch:7.12.0 as well as a docker image in the same host. The docker container is running and host can be accessed by using basic auth.

$ sudo docker container ls
CONTAINER ID   IMAGE                                                  COMMAND                  CREATED             STATUS             PORTS                                                 NAMES
a60bca4121c6   docker.elastic.co/elasticsearch/elasticsearch:7.12.0   "/bin/tini -- /usr/l…"   About an hour ago   Up About an hour   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp   es01
2241695bdfa7   docker.elastic.co/elasticsearch/elasticsearch:7.12.0   "/bin/tini -- /usr/l…"   About an hour ago   Up About an hour   9200/tcp, 9300/tcp                                    es02

$ curl -XGET http://localhost:9200 -u elastic:esbackup-password
{
  "name" : "es01",
  "cluster_name" : "es-docker-cluster",
  "cluster_uuid" : "04EdIVB_QC2il_3iJCarkw",
  "version" : {
    "number" : "7.12.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "78722783c38caa25a70982b5b042074cde5d3b3a",
    "build_date" : "2021-03-18T06:17:15.410153305Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

When I try to execute create-track command using --client-options, I get the ConnectionError mentioned below.

$ sudo docker run elastic/rally create-track --track=mytrack --target-hosts=localhost:9200 --client-options="timeout:60,use_ssl:false,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'esbackup-password'" --indices="my-index-local-*" --output-path=~/tracks

        ____        ____
       / __ \____ _/ / /_  __
      / /_/ / __ `/ / / / / /
     / _, _/ /_/ / / / /_/ /
    /_/ |_|\__,_/_/_/\__, /
                    /____/

    [ERROR] Cannot create-track. ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fdc8ca93220>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fdc8ca93220>: Failed to establish a new connection: [Errno 111] Connection refused).

    Getting further help:
    *********************
    * Check the log files in /rally/.rally/logs for errors.
    * Read the documentation at https://esrally.readthedocs.io/en/2.2.0/.
    * Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
    * Raise an issue at https://github.com/elastic/rally/issues and include the log files in /rally/.rally/logs.

    -------------------------------
    [INFO] FAILURE (took 1 seconds)
    -------------------------------

From the error it is clear that Rally running in docker container is not able to create connection with Elasticsearch running in separate container. I am passing the credentials in --client-options but seems those are not getting picked up.

Not sure, what am I doing wrong here! Any pointers would be appreciated!

Thanks.

[RickBoyd]

Hi @akm,

You already got at the root of the problem, here: your containers cannot (by default) talk to each other, especially on localhost. Your setup would require a docker network to be configured and shared between the Elasticsearch and Rally containers in order for them to communicate.

For an example of how we have Kibana and Elasticsearch containers talking to each other, see our docs. Following that link you can see a network created called "elastic" that is shared among es0* containers and kib01, then the containers refer to each other as hosts by their container names (e.g. http://es01:9200).

You don't state here how your Elasticsearch container is run, but once you have it executing with a network you can run rally with the following (provided you replace the network and container name placeholders):
$ docker run --network <es-network-name> elastic/rally create-track --track=mytrack --target-hosts=<es-container-name>:9200 --client-options="timeout:60,use_ssl:false,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'esbackup-password'" --indices="my-index-local-*" --output-path=~/tracks

Hope that helps!
Rick B

[akm]

Thanks @RickBoyd for the pointers, it really helped. I see I was missing the point of network. It worked perfectly fine after adding --network while running elastic/rally docker and pointing to the es-container-name instead of localhost.