Network settings for ES inside docker in AWS

[boriska70]

I'm running Elasticsearch cluster on EC2 instances while Elasticsearch is running inside docker.
The problem is that sometimes cluster is created as expected and sometimes it doesn't.
When the cluster is not created each one of my 3 nodes starts successfully but says:

[discovery                ] [nodename] waited for 30s and no initial state was set by the discovery
[http                     ] [nodename] publish_address {EC2_privateIP:9200}, bound_addresses {[::]:9200}
[node                     ] [nodename] started
[action.admin.cluster.state] [nodename] no known master node, scheduling a retry
[rest.suppressed          ] /_cluster/state Params: {settings_filter=cloud.key,cloud.account,cloud.aws.access_key,cloud.aws.secret_key,access_key,secret_key,cloud.key,cloud.account,cloud.aws.access_key,cloud.aws.secret_key,access_key,secret_key}
MasterNotDiscoveredException[waited for [30s]]

Elasticsearch version: 2.1

My settings - elasticsearch.yml:

network.host: _ec2_
cloud.aws.access_key: "${AWS_ACCESS_KEY_ID}"
cloud.aws.secret_key: "${AWS_SECRET_ACCESS_KEY}"
cloud.aws.region: "${AWS_REGION}"
discovery.type: "ec2"
discovery.ec2.ping_timeout: "30s"
discovery.zen.ping.multicast.enabled: false
discovery.zen.minimum_master_nodes: 2

My settings - command line arguments in addition to the yml file:
-Des.network.bind_host=0.0.0.0 -Des.cluster.name=${ENVIRON} -Des.node.name=${ENVIRON}-%i;

Docker container exposed ports 9200 and 9300. Network is not accessible from outside and has no public DNS.

There is no network issues between the containers - I can run curl from node1 to node2 and get the succesful response for both docker IP and EC2 IP .

I've checked everything including Elasticsearch settings, AWS settings, security groups, etc. and I do not see the difference between cases when it works and when it does not work.
In any case I guess that my problem is somewhere in the network settings. I have network.host, network.bind_host and network.publish_host. On the other hand I have EC2 private IP and docker IP. So is there any rule what IP should be used eventually in each one of the Elasticsearch network settings?

Thanks!

[boriska70]

The problem was solved by adding discovery.ec2.groups parameter to elasticsearch.yml
From my understanding of https://www.elastic.co/guide/en/elasticsearch/plugins/2.1/cloud-aws-discovery.html this parameter is optional and indeed I have the system that works fine without it. But it looks like it is better to have it configured.
Below please find the example of my elasticsearch.yml - this is not the only possible solution but may be useful for somebody, so I'm publishing it. Note that -Des.network.bind_host=0.0.0.0 -Des.cluster.name=<clustername> -Des.node.name=<nodename> also must be configured, I pass them as a command line arguments.

network.host: _ec2_
bootstrap.mlockall: true
cloud.aws.access_key: "${AWS_ACCESS_KEY_ID}"
cloud.aws.secret_key: "${AWS_SECRET_ACCESS_KEY}"
cloud.aws.region: "${AWS_REGION}"
discovery.type: "ec2"
discovery.ec2.ping_timeout: "30s"
discovery.ec2.groups: [ "elasticsearch_sg_1", "elasticsearch_sg_2" ]
discovery.zen.ping.multicast.enabled: false
discovery.zen.minimum_master_nodes: 2
action.disable_delete_all_indices: true