Configuring Data and Master Node

Elastic Stack Elasticsearch
1

Hello,
I'm building an ELK production cluster consisting of 1 master and 1 (maybe 2) data nodes. I'm following this guide for reference:
https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04
My Elasticsearch version is:
{
"name" : "ubuntu-master",
"cluster_name" : "ubuntu-elk",
"version" : {
"number" : "2.4.0",
"build_hash" : "ce9f0c7394dee074091dd1bc4e9469251181fc55",
"build_timestamp" : "2016-08-29T09:14:17Z",
"build_snapshot" : false,
"lucene_version" : "5.5.2"
},
"tagline" : "You Know, for Search"
}

But I have no preconfigured option for choosing which server is a master or a data node. The below options for reference:

node.master: true
node.data: false

When I input these configurations myself, Elasticsearch falls over and crashes. Any ideas whats going on here?

2

Can you explain how it is crashing? Is there a log or error/exception message in the logs when Elasticsearch fails?

3 
root@elk:/var/log/elasticsearch# service elasticsearch status 
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2016-09-27 09:40:19 BST; 2s ago
     Docs: http://www.elastic.co
  Process: 6472 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -Des.pidfile=${PID_DIR}/elasticsearch.pid -Des.default.path.home=${ES_HOME} -Des.default.path.logs=${LOG_DIR} -Des.default.path.data=${
  Process: 6469 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
 Main PID: 6472 (code=exited, status=1/FAILURE)

Sep 27 09:40:19 elk elasticsearch[6472]: #
Sep 27 09:40:19 elk elasticsearch[6472]: # Use a descriptive name for the node:
Sep 27 09:40:19 elk elasticsearch[6472]: #
Sep 27 09:40:19 elk elasticsearch[6472]:  node.name: ubuntu-master
Sep 27 09:40:19 elk elasticsearch[6472]: #
Sep 27 09:40:19 elk elasticsearch[6472]: # Add custom attributes to the node:
Sep 27 09:40:19 elk elasticsearch[6472]: #
Sep 27 09:40:19 elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Sep 27 09:40:19 elk systemd[1]: elasticsearch.service: Unit entered failed state.
Sep 27 09:40:19 elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

the only thing in the log is this:

[2016-09-27 09:40:18,315][INFO ][node                     ] [ubuntu-master] stopping ...
[2016-09-27 09:40:18,580][INFO ][node                     ] [ubuntu-master] stopped
[2016-09-27 09:40:18,581][INFO ][node                     ] [ubuntu-master] closing ...
[2016-09-27 09:40:18,588][INFO ][node                     ] [ubuntu-master] closed

Edit: This is the build number and version

Version: 2.4.0, Build: ce9f0c7/2016-08-29T09:14:17Z, JVM: 1.7.0_80

4

Depending on where you have the configuration directory, can you try running ES manually (to see if it logs something else to the console that isn't making it to the logs), so something like:

bin/elasticsearch -Dpath.config=/etc/elasticsearch

Replace /etc/elasticsearch with the directory where your configuration (elasticsearch.yml) file exists

Hopefully that will show more information about why Elasticsearch is not starting.

5

When ES is run manually, The log file produces this:

[2016-09-27 10:01:24,401][INFO ][node                     ] [ubuntu-master] version[2.4.0], pid[6783], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-09-27 10:01:24,402][INFO ][node                     ] [ubuntu-master] initializing ...
[2016-09-27 10:01:25,374][INFO ][plugins                  ] [ubuntu-master] modules [lang-groovy, reindex, lang-expression], plugins [], sites []
[2016-09-27 10:01:25,395][INFO ][env                      ] [ubuntu-master] using [1] data paths, mounts [[/ (/dev/mapper/elk--vg-root)]], net usable_space [36.9gb], net total_space [44.6gb], spins? [possibly], types [ext4]
[2016-09-27 10:01:25,395][INFO ][env                      ] [ubuntu-master] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-09-27 10:01:27,443][INFO ][node                     ] [ubuntu-master] initialized
[2016-09-27 10:01:27,443][INFO ][node                     ] [ubuntu-master] starting ...
[2016-09-27 10:01:27,484][INFO ][transport                ] [ubuntu-master] publish_address {192.168.70.224:9300}, bound_addresses {192.168.70.224:9300}
[2016-09-27 10:01:27,495][INFO ][discovery                ] [ubuntu-master] ubuntu-elk/XMehwcG5SAulQXIR2yAYoQ
[2016-09-27 10:01:30,528][INFO ][cluster.service          ] [ubuntu-master] new_master {ubuntu-master}{XMehwcG5SAulQXIR2yAYoQ}{192.168.70.224}{192.168.70.224:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-09-27 10:01:30,553][INFO ][http                     ] [ubuntu-master] publish_address {192.168.70.224:9200}, bound_addresses {192.168.70.224:9200}
[2016-09-27 10:01:30,553][INFO ][node                     ] [ubuntu-master] started
[2016-09-27 10:01:30,736][INFO ][gateway                  ] [ubuntu-master] recovered [3] indices into cluster_state
[2016-09-27 10:01:32,082][INFO ][cluster.routing.allocation] [ubuntu-master] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).
[2016-09-27 10:02:01,153][INFO ][node                     ] [ubuntu-master] stopping ...
[2016-09-27 10:02:01,445][INFO ][node                     ] [ubuntu-master] stopped
[2016-09-27 10:02:01,445][INFO ][node                     ] [ubuntu-master] closing ...
[2016-09-27 10:02:01,450][INFO ][node                     ] [ubuntu-master] closed

running the command in terminal produces this:

root@elk:/usr/share/elasticsearch/bin# sudo ./elasticsearch -Dpath.config=/etc/elasticsearch/
Exception in thread "main" ElasticsearchException[Failed to load logging configuration]; nested: NoSuchFileException[/usr/share/elasticsearch/config];
Likely root cause: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
	at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
	at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
	at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
	at java.nio.file.Files.readAttributes(Files.java:1686)
	at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:109)
	at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:69)
	at java.nio.file.Files.walkFileTree(Files.java:2602)
	at org.elasticsearch.common.logging.log4j.LogConfigurator.resolveConfig(LogConfigurator.java:142)
	at org.elasticsearch.common.logging.log4j.LogConfigurator.configure(LogConfigurator.java:103)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:259)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.

This all with the master.node and data.node added to the config.

6

Shouldn't it be -Des.path.config instead?

7

That returns the exact same issues as above

8

Sorry. It should be -Des.path.conf

9

That one works, it displayed a new error:

; line: 98, column: 1]]; nested: ParserException[expected '<document start>', but found BlockMappingStart
 in 'reader', line 98, column 1:
    node.master: true
    ^
];
Likely root cause: expected '<document start>', but found BlockMappingStart
 in 'reader', line 98, column 1:
    node.master: true
    ^

	at org.yaml.snakeyaml.parser.ParserImpl$ParseDocumentStart.produce(ParserImpl.java:225)
	at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:158)
	at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:168)
	at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:340)
	at org.elasticsearch.common.xcontent.json.JsonXContentParser.nextToken(JsonXContentParser.java:53)
	at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:72)
	at org.elasticsearch.common.settings.loader.XContentSettingsLoader.load(XContentSettingsLoader.java:45)
	at org.elasticsearch.common.settings.loader.YamlSettingsLoader.load(YamlSettingsLoader.java:46)
	at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1080)
	at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1067)
	at org.elasticsearch.node.internal.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:88)
	at org.elasticsearch.bootstrap.Bootstrap.initialSettings(Bootstrap.java:218)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:257)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)
Refer to the log for complete error details.

However, it has stopped logging since the first time I ran the command dakrone recommended.

10

Your config/elasticsearch.yml file is incorrect.
Check that you don't have any space before node.master. It should be something like:

node.master: true

and not

 node.master: true
11

All the above errors are still in place. "Failed to laod logging configuration" is common across all these errors, I havent changed the logging.yml file at all so its all default

Edit: It only begins logging once I run it via a service "sudo service elasticsearch start". Running it manually causes all the above issues.

12

Do you have a logging.yml file in /etc/elasticsearch?

That being said may be you could download a fresh tar.gz version of elasticsearch and try to run it on your machine manually to see if your initial issue could be related to your hardware or whatever?

13

Yeah I have that file in that directory, I freshly installed ES and this is still occuring :\