I've setup transport and http layer security but my two nodes can't detect eachother?

I'm trying to secure my cluster through the tutorial in the docs. I have two EC2 instances running. I have installed elasticsearch on both of them. I've setup transport layer and http layer security. I have started the elasticsearch service via systemctl and systemctl status shows they are up and running.

From the EC2 that is running my master-node-1 I have typed the following curl command and the output is shown...

ec2-user@ip-172-32-56-218
[~] > curl -u elastic -k -XGET 'https://172.32.56.218:9200/_cluster/health?pretty'
Enter host password for user 'elastic':
{
  "cluster_name" : "elasticsearch-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 6,
  "active_shards" : 6,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 1,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 85.71428571428571
}

You can see there's only one node!? Why isn't the other node (data-node-1) detected? The following are my two elasticsearch.yml configurations for each node.

master-node-1

cluster.name: elasticsearch-cluster                                              
node.name: master-node-1                                                         
path.data: /var/lib/elasticsearch                                                
path.logs: /var/log/elasticsearch                                                
bootstrap.memory_lock: true                                                      
network.host: 172.32.56.218                                                      
http.port: 9200                                                                  
discovery.seed_hosts: ["172.32.57.175:9200"]                                     
xpack.security.enabled: true                                                     
xpack.security.transport.ssl.enabled: true                                       
xpack.security.transport.ssl.verification_mode: full                             
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/master-node-1.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/master-node-1.p12
xpack.security.http.ssl.enabled: true                                            
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/http.p12         
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/http.p12       
xpack.security.http.ssl.client_authentication: optional                          
xpack.security.authc.realms.pki.pki1:                                            
  enabled: true 

data-node-1

cluster.name: elasticsearch-cluster                                              
node.name: data-node-1                                                           
path.data: /var/lib/elasticsearch                                                
path.logs: /var/log/elasticsearch                                                
bootstrap.memory_lock: true                                                      
network.host: 172.32.57.175                                                      
http.port: 9200                                                                  
discovery.seed_hosts: ["172.32.56.218:9200"]                                     
xpack.security.enabled: true                                                     
xpack.security.transport.ssl.enabled: true                                       
xpack.security.transport.ssl.verification_mode: full                             
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/certs/data-node-1.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/certs/data-node-1.p12                                                      
xpack.security.http.ssl.enabled: true                                            
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/certs/http.p12         
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/certs/http.p12       
xpack.security.http.ssl.client_authentication: optional                          
xpack.security.authc.realms.pki.pki1:                                            
  enabled: true

From the other EC2 instance where data-node-1 is running I ran the following commands to see what would happen. I keep getting security_exception errors. Not sure if this is related but could use some tips on what to try.

ec2-user@ip-172-32-57-175
[~] > curl -k -u elastic -XGET 'https://172.32.57.175:9200/_license?pretty'
Enter host password for user 'elastic':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "failed to authenticate user [elastic]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "failed to authenticate user [elastic]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

ec2-user@ip-172-32-57-175
[~] > curl -k -XGET 'https://172.32.57.175:9200/_license?pretty'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "missing authentication credentials for REST request [/_license?pretty]",
        "header" : {
          "WWW-Authenticate" : [
            "Bearer realm=\"security\"",
            "ApiKey",
            "Basic realm=\"security\" charset=\"UTF-8\""
          ]
        }
      }
    ],
    "type" : "security_exception",
    "reason" : "missing authentication credentials for REST request [/_license?pretty]",
    "header" : {
      "WWW-Authenticate" : [
        "Bearer realm=\"security\"",
        "ApiKey",
        "Basic realm=\"security\" charset=\"UTF-8\""
      ]
    }
  },
  "status" : 401
}

I have 3 comments:

  • with your current configuration, except if you're passing environment variables, both nodes are master eligible
  • the setting discovery.seed_hosts requires the port 9300
  • you're missing the initial_master_nodes setting
  1. Please add to both nodes:
cluster.initial_master_nodes: 
   - master-node-1
   - data-node-1

Do not use the IPs on cluster.initial_master_nodes.

  1. Change the port on discovery.seed_hosts to 9300.

Once you've done this please restart and if you still have issues, share the logs.

More info at https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html

@Luca_Belluccini

Are you saying that I can no longer use 9200 now? I take it that means http.port needs to be changed to 9300 as well then?

Each Elasticsearch node requires one port to talk to each other (Transport, 9300) and the http port (9200)

I think this isn't right. From the docs:

You should not use this setting when restarting a cluster or adding a new node to an existing cluster.

Since the OP has already formed a cluster and is trying to add a data-only node to it, they should not be using this setting.

I think the problem is the port in discovery.seed_hosts and the fact that the data node doesn't have node.master: false.

Thank you @DavidTurner
Right as @syost requested a data only node.

Depending on what we want to obtain, the master-node-1 should have node.data: false to be a master only node.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.