Discovery-ec2 plugin always tries to ping localhost / never finds the nodes that it should

DavidTurner · December 11, 2018, 8:08pm

The pinging of localhost is a diversion, I think. That's the default behaviour if discovery.zen.ping.unicast.hosts is unset. You can suppress it by setting it to the empty list instead:

discovery.zen.ping.unicast.hosts: []

The real issue lies between these two lines:

[2018-12-11T00:57:41,477][TRACE][o.e.d.e.AwsEc2UnicastHostsProvider] [ip-172-25-34-206] building dynamic unicast discovery nodes...
[2018-12-11T00:57:41,478][DEBUG][o.e.d.e.AwsEc2UnicastHostsProvider] [ip-172-25-34-206] using dynamic transport addresses []

The first line is reported here:

github.com

elastic/elasticsearch/blob/baf33b31626f13521d4862d0b700c7c6d728bb6f/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider.java#L120


    // NOTE: we don't filter by security group during the describe instances request for two reasons:
    // 1. differences in VPCs require different parameters during query (ID vs Name)
    // 2. We want to use two different strategies: (all security groups vs. any security groups)
    descInstances = SocketAccess.doPrivileged(() -> clientReference.client().describeInstances(buildDescribeInstancesRequest()));
} catch (final AmazonClientException e) {
    logger.info("Exception while retrieving instance list from AWS API: {}", e.getMessage());
    logger.debug("Full exception:", e);
    return dynamicHosts;
}


logger.trace("building dynamic unicast discovery nodes...");
for (final Reservation reservation : descInstances.getReservations()) {
    for (final Instance instance : reservation.getInstances()) {
        // lets see if we can filter based on groups
        if (!groups.isEmpty()) {
            final List<GroupIdentifier> instanceSecurityGroups = instance.getSecurityGroups();
            final List<String> securityGroupNames = new ArrayList<>(instanceSecurityGroups.size());
            final List<String> securityGroupIds = new ArrayList<>(instanceSecurityGroups.size());
            for (final GroupIdentifier sg : instanceSecurityGroups) {
                securityGroupNames.add(sg.getGroupName());
                securityGroupIds.add(sg.getGroupId());

After that Elasticsearch loops through all the instances returned by the DescribeInstances call and logs various things at various points, before finally logging using dynamic transport addresses .... That there is nothing logged between these two lines suggests that nothing is returned by EC2.

Can you construct an analogous DescribeInstances call and verify that the right instances are genuinely returned? The relevant code is here:

github.com

elastic/elasticsearch/blob/baf33b31626f13521d4862d0b700c7c6d728bb6f/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2UnicastHostsProvider.java#L200-L221


private DescribeInstancesRequest buildDescribeInstancesRequest() {
    final DescribeInstancesRequest describeInstancesRequest = new DescribeInstancesRequest()
        .withFilters(
            new Filter("instance-state-name").withValues("running", "pending")
        );


    for (final Map.Entry<String, List<String>> tagFilter : tags.entrySet()) {
        // for a given tag key, OR relationship for multiple different values
        describeInstancesRequest.withFilters(
            new Filter("tag:" + tagFilter.getKey()).withValues(tagFilter.getValue())
        );
    }


    if (!availabilityZones.isEmpty()) {
        // OR relationship amongst multiple values of the availability-zone filter
        describeInstancesRequest.withFilters(
            new Filter("availability-zone").withValues(availabilityZones)
        );
    }

This file has been truncated. show original

Topic		Replies	Views
Node seems not to be discovered on EC2 network Elasticsearch	4	654	July 8, 2020
ec2 discovery Elasticsearch	7	401	June 23, 2011
EC2 Configuration Elasticsearch	12	456	July 6, 2017
Cluster discovery on Amazon EC2 problem - need urgent help Elasticsearch	11	1684	July 6, 2017
EC2 discovery issue Elasticsearch	30	1832	July 5, 2017

Discovery-ec2 plugin always tries to ping localhost / never finds the nodes that it should

Related topics