Problems creating a single node cluster

Hi!

I've managed to get Elastic 7.6 running on Ubuntu 18.04 LTS in AWS, I have access to it via a REST client using port tunnelling in a SSH connection, and while it is accessible (returning the standard "status" message from curl -XGET 'localhost:8881/'), there are errors, which — after discussing with someone more qualified than myself — appear related to Elastic struggling with nodes:

[2020-02-19T09:23:35,746][WARN ][o.e.c.c.ClusterFormationFailureHelper] [ip-172-31-14-190] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [52.2.0.8] to bootstrap a cluster: have discovered [{ip-172-31-14-190}{ECULpkNnST6lEdrAJfp5Yg}{n3DpWCldST6GxNn0sHUAHw}{localhost}{127.0.0.1:9300}{dilm}{ml.machine_memory=4086390784, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{ip-172-31-14-190}{ECULpkNnST6lEdrAJfp5Yg}{n3DpWCldST6GxNn0sHUAHw}{localhost}{127.0.0.1:9300}{dilm}{ml.machine_memory=4086390784, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0

So, as advised, I edited: "elasticsearch.yml" and added:

discovery.type: single-node

… but get the following errors after running the command journalctl -xe:

-- The result is RESULT.
Feb 19 09:50:13 ip-172-31-14-190 sudo[9133]: ubuntu : TTY=pts/0 ; PWD=/etc/elasticsearch ; USER=root ; COMMAND=/bin/nano elasticsearch.yml
Feb 19 09:50:13 ip-172-31-14-190 sudo[9133]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
Feb 19 09:50:16 ip-172-31-14-190 sshd[9135]: Received disconnect from [IP_ADDRESS] port 60385:11: [preauth]
Feb 19 09:50:16 ip-172-31-14-190 sshd[9135]: Disconnected from authenticating user root [IP_ADDRESS] port 60385 [preauth]
Feb 19 09:50:40 ip-172-31-14-190 sudo[9133]: pam_unix(sudo:session): session closed for user root
Feb 19 09:53:57 ip-172-31-14-190 sshd[9138]: Received disconnect from [IP_ADDRESS] port 52178:11: [preauth]
Feb 19 09:53:57 ip-172-31-14-190 sshd[9138]: Disconnected from authenticating user root [IP_ADDRESS] port 52178 [preauth]
Feb 19 09:55:02 ip-172-31-14-190 sudo[9140]: ubuntu : TTY=pts/0 ; PWD=/etc/elasticsearch ; USER=root ; COMMAND=/bin/nano elasticsearch.yml
Feb 19 09:55:02 ip-172-31-14-190 sudo[9140]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
Feb 19 09:55:04 ip-172-31-14-190 sshd[9142]: Received disconnect from [IP_ADDRESS] port 39241:11: Bye Bye [preauth]
Feb 19 09:55:04 ip-172-31-14-190 sshd[9142]: Disconnected from [IP_ADDRESS] port 39241 [preauth]
Feb 19 09:55:53 ip-172-31-14-190 sudo[9140]: pam_unix(sudo:session): session closed for user root
Feb 19 09:55:58 ip-172-31-14-190 polkitd(authority=local)[866]: Registered Authentication Agent for unix-process:9145:6837580 (system bus name :1.73 [/usr/bin/pkttyagent --notify-fd 5 --fallback], obj
Feb 19 09:56:02 ip-172-31-14-190 polkitd(authority=local)[866]: Unregistered Authentication Agent for unix-process:9145:6837580 (system bus name :1.73, object path /org/freedesktop/PolicyKit1/Authenti
Feb 19 09:56:02 ip-172-31-14-190 polkitd(authority=local)[866]: Operator of unix-process:9145:6837580 FAILED to authenticate to gain authorization for action org.freedesktop.systemd1.manage-units for
Feb 19 09:56:05 ip-172-31-14-190 sudo[9153]: ubuntu : TTY=pts/0 ; PWD=/etc/elasticsearch ; USER=root ; COMMAND=/bin/systemctl restart elasticsearch.service
Feb 19 09:56:05 ip-172-31-14-190 sudo[9153]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
Feb 19 09:56:05 ip-172-31-14-190 systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu

-- Unit elasticsearch.service has begun starting up.
Feb 19 09:56:06 ip-172-31-14-190 elasticsearch[9156]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 19 09:56:15 ip-172-31-14-190 systemd-timesyncd[538]: Network configuration changed, trying to establish connection.
Feb 19 09:56:15 ip-172-31-14-190 systemd-timesyncd[538]: Synchronized to time server [IP_ADDRESS] (ntp.ubuntu.com).
Feb 19 09:56:24 ip-172-31-14-190 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 09:56:24 ip-172-31-14-190 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
Feb 19 09:56:24 ip-172-31-14-190 systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: Enterprise open source support | Ubuntu

-- Unit elasticsearch.service has failed.

-- The result is RESULT.
Feb 19 09:56:24 ip-172-31-14-190 sudo[9153]: pam_unix(sudo:session): session closed for user root
Feb 19 09:56:39 ip-172-31-14-190 sudo[9299]: ubuntu : TTY=pts/0 ; PWD=/etc/elasticsearch ; USER=root ; COMMAND=/bin/nano elasticsearch.yml
Feb 19 09:56:39 ip-172-31-14-190 sudo[9299]: pam_unix(sudo:session): session opened for user root by ubuntu(uid=0)
Feb 19 09:57:19 ip-172-31-14-190 sudo[9299]: pam_unix(sudo:session): session closed for user root

I don't remember having these problems with Elastic 6.x in production, or installing 7.x on the development Mac, so I'm at a loss here.

At this point, it's not as important to me what the problem is as it is getting Elastic up, running, and stable.

Unfortunately the journalctl logs are not really of any use here beyond telling us that Elasticsearch started and then stopped, and whatever needs adjusting will be detailed in the Elasticsearch logs. You have shared a single line from Elasticsearch logs from before setting discovery.type: single-node (the message from o.e.c.c.ClusterFormationFailureHelper); can you share the same logs from after adjusting this setting?

Hi @DavidTurner, I ran the command sudo systemctl start elasticsearch and got:

Job for elasticsearch.service failed because the control process exited with error code.

See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

In a separate session, I had the command running: tail -f elasticsearch.log but nothing appeared when Elastic failed.

What else would I need to do?

Hi @waynesmallman.

this ' curl -XGET 'localhost:8881/' is working with AWS machine?

or you are are running the same curl command out of aws machine?

case 01:- if you are able to run that command with aws machine then fine if not then some issue with your elastic.yml file
case :02 if you are running this curl command out of aws machine then you need to map host with localhost of aws machine.

Please check ?

Thanks
HadoopHelp

Hi @rameshkr1994, as I said the curl command is working — I swapped the port number based on advice regarding the ELK stack and how to secure it.

So, to be clear — running the curl command works both from the REST client and on Ubuntu.

Hi @waynesmallman .

may be i am not getting your issue nut i tried to understand as below :- command getting error in culr:-

Hi @rameshkr1994, I don't think you do understand — please read the original message again, as it answers the questions you're asking me.

As with your previous thread, it's hard to give any concrete suggestions without seeing the Elasticsearch logs. I don't really understand why you cannot see them, sorry, but without them we're kinda stuck. I tried again to reproduce this (see my post in your earlier thread, a clean installation on a brand-new Ubuntu 18.04 instance) and sudo tail -f /var/log/elasticsearch/elasticsearch.log shows the logs we need.

Have you adjusted the logging config from the default? If so, can you put it back to how it was? There's certainly been some customisation because this node must discover master-eligible nodes [52.2.0.8] to bootstrap a cluster is not how an empty installation behaves.

Hi @DavidTurner, I am now able to see the logs (I had to add "ubuntu" to the "elasticsearch" group, which Elastic itself created).

I've been running more tests:

ubuntu@ip-172-31-14-190:~$ systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-02-19 19:50:06 UTC; 48min ago
Docs: http://www.elastic.co
Main PID: 10812 (java)
Tasks: 65 (limit: 4647)
CGroup: /system.slice/elasticsearch.service
├─10812 /usr/share/elasticsearch/jdk/bin/java -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=
└─10929 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Feb 19 19:49:14 ip-172-31-14-190 systemd[1]: Starting Elasticsearch...
Feb 19 19:49:15 ip-172-31-14-190 elasticsearch[10812]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 19 19:50:06 ip-172-31-14-190 systemd[1]: Started Elasticsearch.

And it appears that Elastic is running:

ubuntu@ip-172-31-14-190:~$ curl -XGET 'localhost:8881/'
{
"name" : "ip-172-31-14-190",
"cluster_name" : "under-cloud",
"cluster_uuid" : "na",
"version" : {
"number" : "7.5.2",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
"build_date" : "2020-01-15T12:11:52.313576Z",
"build_snapshot" : false,
"lucene_version" : "8.3.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

… and — as explained — I'm able to run the Curl request from both the REST client and from within Ubuntu.

But note: "cluster_uuid" : "_na_"

When I run a PUT request for: "http://localhost:9000/notes_en_v2" which has a JSON object to create an index, one that I've used on both Elastic 6.x and 7.x in development, it hangs on this staging environment, and I get the following in Insomnia:

{
"error": {
"root_cause": [
{
"type": "master_not_discovered_exception",
"reason": null
}
],
"type": "master_not_discovered_exception",
"reason": null
},
"status": 503
}

… but there's no corresponding error in the logs.

David, something's wrong with this installation of Elastic, and I have followed the instructions to the letter, as I have done with the last umpteen installations of Elastic.

Great. Can you share them here? Ideally everything from the node starting up for at least 5 minutes.

In the end, I had to comment out the cluster.initial_master_nodes option in the "elasticsearch.yml" file.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.