Talk to server... ERROR Connection marked as failed because the onConnect callback failed: invalid license found, requires a basic or a valid trial l icense and received Open sour

Elastic Stack Elasticsearch
1

Hi,

I am still facing this license problem to one of my ELK nodes.

elasticsearch: http://itkvmxh01.emea.nsn-net.net:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 151.98.17.61
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... ERROR Connection marked as failed because the onConnect callback failed: invalid license found, requires a basic or a valid trial l
icense and received Open source

talk to server... ERROR Connection marked as failed because the onConnect callback failed: invalid license found, requires a basic or a valid trial l
icense and received Open source.

This is the repo that i have installed the elasticsearc on the node:

[elasticsearch-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

And here is my elasticsearch.yml file.
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: xh-elastic-cluster-1
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
# node.name: xh-ita-elastic-1
node.name: xh-it-elastic-2
cluster.initial_master_nodes:
- xh-gr-elastic-1
- xh-fr-elastic-1
- xh-it-elastic-1
discovery.seed_hosts:
- 10.158.67.175
- 10.159.166.9
- 10.158.67.107
- 151.98.17.60
- 151.98.17.61
- 135.238.239.48
- 135.238.239.132
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
xpack.license.self_generated.type: basic
xpack.security.enabled: false
#node.master: true
#node.data: false
#node.ingest: false
#cluster.remote.connect: false
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#network.bind_host: 0.0.0.0
#network.publish_host: 0.0.0.0
http.port: 9200
# transport.host: localhost
transport.host: 0.0.0.0
transport.tcp.port: 9300
#network.host: localhost
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["10.158.67.175", "10.159.166.9", "10.158.67.107", "151.98.17.60", "151.98.17.61", "135.238.239.48", "135.238.239.13
2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

2

Do you have any idea why i am facing this error?

3

Is it possible that you have installed the pure OSS distribution of Elasticsearch. Your yum configuration does not look like it, so the other possibility could be, that you are running another unofficial package of elasticsearch.

Can you paste the output of these queries

GET /

GET /_license

GET _license/basic_status

GET _license/trial_status
4 
{
  "name" : "xh-gr-elastic-1",
  "cluster_name" : "xh-elastic-cluster-1",
  "cluster_uuid" : "ap97EWHPRNOeejHqqSYP8w",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
5 
{
  "license" : {
    "status" : "active",
    "uid" : "9b8384d7-06a0-47d8-9b89-698e0724431a",
    "type" : "basic",
    "issue_date" : "2019-09-06T13:03:14.672Z",
    "issue_date_in_millis" : 1567774994672,
    "max_nodes" : 1000,
    "issued_to" : "xh-elastic-cluster-1",
    "issuer" : "elasticsearch",
    "start_date_in_millis" : -1
  }
}
6 
{
  "eligible_to_start_basic" : false
}
7 
{
  "eligible_to_start_trial" : true
}
8

The error message at the top mentions

http://itkvmxh01.emea.nsn-net.net:9200

however the node name in your configuration does not match this? Does the beat you are trying to start connect to another elasticsearch instance?

10

I send you the output of the GET from the Dev Tools tab inside the Kibana not from the node that is not connected to to the cluster.

Now i run them inside the node that is not connected.

7
7.PNG1515×510 19.6 KB

11 
[root@itkvmxh01 ~]# curl -XGET "http://localhost:9200/"

{
  "name" : "xh-it-elastic-2",
  "cluster_name" : "xh-elastic-cluster-1",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.5.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e9ccaed468e2fac2275a3761849cbee64b39519f",
    "build_date" : "2019-11-26T01:06:52.518245Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@itkvmxh01 ~]# curl -XGET "http://localhost:9200/_license"
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
[root@itkvmxh01 ~]# curl -XGET "http://localhost:9200/_license/basic_status"

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
[root@itkvmxh01 ~]# curl -XGET "http://localhost:9200/_license/trial_status"

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
12

If you see from my last commands on the node. The node cannot get the cluster uuid.

13 
[root@itkvmxh01 ~]# metricbeat test output

elasticsearch: http://atdevxhv03.emea.nsn-net.net:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.158.67.175
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.5.0
elasticsearch: http://it017060.it.alcatel-lucent.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 151.98.17.60
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.5.0
elasticsearch: http://frelkxh01.fr.alcatel-lucent.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 135.238.239.48
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.5.0
elasticsearch: http://atkvmxh01.emea.nsn-net.net:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.159.166.9
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.5.0
elasticsearch: http://frkvmxh01.fr.alcatel-lucent.com:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 135.238.239.132
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.5.0
elasticsearch: http://atdevxhv04.nsn-intra.net:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.158.67.107
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
  version: 7.5.0
elasticsearch: http://itkvmxh01.emea.nsn-net.net:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 151.98.17.61
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... ERROR Connection marked as failed because the onConnect callback failed: invalid license found, requires a basic or a valid trial l
icense and received Open source

Yes the beats are connected to all the other nodes on the ELK cluster. The problem is that this node cannot connected to the ELK cluster. The installation procedure for the elasticsearch to this node is exactly the same as the other nodes.

14

it seems the last node has not joined the cluster. Checking the logs of that node would be the next step.

15

The logs of the elasticsearch you mean?

16 
[2019-12-11T12:11:04,735][INFO ][o.e.c.c.JoinHelper       ] [xh-it-elastic-2] failed to join {xh-fr-elastic-1}{CBiU9s_BSai2oBfVeZjwiQ}{8uq4Dht3Tzeb9oKibUaT8w}{135.238.239.48}{135.238.239.48:9300}{dilm}{ml.machine_memory=16654974976, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={xh-it-elastic-2}{6uKI_xAvSuu7R4NjkQJNFA}{uN-e-hsiSESxzCPg3s5WuA}{151.98.14.14}{151.98.14.14:9300}{dilm}{ml.machine_memory=269931491328, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [xh-fr-elastic-1][135.238.239.48:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.ConnectTransportException: [xh-it-elastic-2][151.98.14.14:9300] connect_exception

Hi again. I am getting these logs when i restart the elasticsearch service on this node.

17

[2019-12-11T12:11:03,161][WARN ][o.e.c.c.ClusterFormationFailureHelper] [xh-it-elastic-2] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [xh-gr-elastic-1, xh-fr-elastic-1, xh-it-elastic-1] to bootstrap a cluster: have discovered [{xh-it-elastic-2}{6uKI_xAvSuu7R4NjkQJNFA}{uN-e-hsiSESxzCPg3s5WuA}{151.98.14.14}{151.98.14.14:9300}{dilm}{ml.machine_memory=269931491328, xpack.installed=true, ml.max_open_jobs=20}, {xh-gr-elastic-1}{EmyFlH3zS3KkvJptV02pMQ}{gzuvyM9uSg-AaV34aYZyWg}{10.158.67.175}{10.158.67.175:9300}{dilm}{ml.machine_memory=16654729216, ml.max_open_jobs=20, xpack.installed=true}, {xh-gr-elastic-2}{46jaKQPSRC2drwy-bAfhxw}{tMU977R8THyw5bCf8re3kQ}{10.159.166.9}{10.159.166.9:9300}{dilm}{ml.machine_memory=269930561536, ml.max_open_jobs=20, xpack.installed=true}, {xh-gr-elastic-3}{bLV-av6fSAWBTep4FKAqWA}{jAkjiFSSQJOEZmq3nwx9mA}{10.158.67.107}{10.158.67.107:9300}{dilm}{ml.machine_memory=17179332608, ml.max_open_jobs=20, xpack.installed=true}, {xh-it-elastic-1}{g9M0MIndTBW62pSrEdktzA}{Fph4y6aVQE69CuNS9SKDuA}{151.98.17.60}{151.98.17.60:9300}{dilm}{ml.machine_memory=34359738368, ml.max_open_jobs=20, xpack.installed=true}, {xh-fr-elastic-1}{CBiU9s_BSai2oBfVeZjwiQ}{8uq4Dht3Tzeb9oKibUaT8w}{135.238.239.48}{135.238.239.48:9300}{dilm}{ml.machine_memory=16654974976, ml.max_open_jobs=20, xpack.installed=true}, {xh-fr-elastic-2}{obT4-IMdTU6kOAHKVFNxzA}{KEkLq7X-SlyAghnCKkG3cQ}{135.238.239.132}{135.238.239.132:9300}{dilm}{ml.machine_memory=269930713088, ml.max_open_jobs=20, xpack.installed=true}]; discovery will continue using [10.158.67.175:9300, 10.159.166.9:9300, 10.158.67.107:9300, 151.98.17.60:9300, 151.98.17.61:9300, 135.238.239.48:9300, 135.238.239.132:9300] from hosts providers and [{xh-it-elastic-2}{6uKI_xAvSuu7R4NjkQJNFA}{uN-e-hsiSESxzCPg3s5WuA}{151.98.14.14}{151.98.14.14:9300}{dilm}{ml.machine_memory=269931491328, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 6874, last-accepted version 0 in term 0

18

Caused by: java.io.IOException: connection timed out: 151.98.14.14/151.98.14.14:9300

19

Do you have any idea about the output of the logs that i have sent? It seems that it cannot connect to the cluster something like java timeout on the connection

20

can that host reach the elasticsearch data port of other node or is there maybe any firewall inbetween?

21

They can reach each other firewalls are off on all the nodes