Elasticsearch seems to ignore settings in yaml file

Elastic Stack Elasticsearch
1

Hi,
I have an ES cluster on a single machine consisting of five nodes: master, data1, data2, data3 and ingest. After upgrading from 5.5.1 to 6.8.1, I have managed to get the master node up and running, however, other nodes won't start.

Master node's elasticsearch.yml:

cluster.name: elastic
node.name: master
node.master: true
node.data: false
node.ingest: false
node.max_local_storage_nodes: 5

path.data: /data/elasticsearch/data_master
path.logs: /var/log/elasticsearch/master

bootstrap.memory_lock: true

network.host: 192.168.1.52
http.port: 9200
transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["192.168.1.52:9301","192.168.1.52:9302","192.168.1.52:9303","192.168.1.52:9304"]

Ingest node's elasticsearch.yml:

cluster.name: elastic
node.name: ingest
node.master: false
node.data: false
node.ingest: true
node.max_local_storage_nodes: 5

path.data: /data/elasticsearch/data_ingest
path.logs: /var/log/elasticsearch/ingest

bootstrap.memory_lock: true

network.host: 192.168.1.52
http.port: 9204
transport.tcp.port: 9304

discovery.zen.ping.unicast.hosts: ["192.168.1.52:9300"]

Now, after trying to run the ingest node while the master is already running, I have noticed that both nodes write the logs to /var/log/elasticsearch/elastic.log despite having their log path defined in their respective yaml files.

This is what ingest logs after failing to start:

[2019-07-07T10:53:02,214][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1]
[2019-07-07T10:53:02,226][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.jar:6.8.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
	... 6 more

Note the line ending with "multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?". node.max_local_storage_nodes is set to 5 in yaml, but seems to be ignored too.

I'm really confused as to why some settings seem to be ignored. Am I missing something?

Thanks!

2

I wonder if elasticsearch reads the configuration from another directory. Could you change the node name or put in your configuration file a bad content so it's not a yml anymore?

Note that in the future this won't be supported anymore and that you should have a data dir for each node (so a config dir for each node).

3

You're right. The yaml files of respective nodes are in /etc/<node_name>/elasticsearch.yml, but Elasticsearch reads the configuration from /etc/elasticsearch/elasticsearch.yml.

However, this shouldn't be the case, since all of my nodes have their own config directories, defined in their respective service scripts. Here is an example of elasticsearch_master.service :

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/master
Environment=DATA_DIR=/var/lib/elasticsearch/master
Environment=LOG_DIR=/var/log/elasticsearch/master
Environment=ES_PATH_CONF=/etc/master
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.