Elasticsearch seems to ignore settings in yaml file

elasticFox · July 7, 2019, 12:18pm

Hi,
I have an ES cluster on a single machine consisting of five nodes: master, data1, data2, data3 and ingest. After upgrading from 5.5.1 to 6.8.1, I have managed to get the master node up and running, however, other nodes won't start.

Master node's elasticsearch.yml:

cluster.name: elastic
node.name: master
node.master: true
node.data: false
node.ingest: false
node.max_local_storage_nodes: 5

path.data: /data/elasticsearch/data_master
path.logs: /var/log/elasticsearch/master

bootstrap.memory_lock: true

network.host: 192.168.1.52
http.port: 9200
transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["192.168.1.52:9301","192.168.1.52:9302","192.168.1.52:9303","192.168.1.52:9304"]

Ingest node's elasticsearch.yml:

cluster.name: elastic
node.name: ingest
node.master: false
node.data: false
node.ingest: true
node.max_local_storage_nodes: 5

path.data: /data/elasticsearch/data_ingest
path.logs: /var/log/elasticsearch/ingest

bootstrap.memory_lock: true

network.host: 192.168.1.52
http.port: 9204
transport.tcp.port: 9304

discovery.zen.ping.unicast.hosts: ["192.168.1.52:9300"]

Now, after trying to run the ingest node while the master is already running, I have noticed that both nodes write the logs to /var/log/elasticsearch/elastic.log despite having their log path defined in their respective yaml files.

This is what ingest logs after failing to start:

[2019-07-07T10:53:02,214][ERROR][o.e.b.Bootstrap          ] [node-1] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.1.jar:6.8.1]
[2019-07-07T10:53:02,226][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.1.jar:6.8.1]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
	at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:300) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.1.jar:6.8.1]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.1.jar:6.8.1]
	... 6 more

Note the line ending with "multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?". node.max_local_storage_nodes is set to 5 in yaml, but seems to be ignored too.

I'm really confused as to why some settings seem to be ignored. Am I missing something?

Thanks!

dadoonet · July 7, 2019, 1:59pm

I wonder if elasticsearch reads the configuration from another directory. Could you change the node name or put in your configuration file a bad content so it's not a yml anymore?

Note that in the future this won't be supported anymore and that you should have a data dir for each node (so a config dir for each node).

elasticFox · July 7, 2019, 3:27pm

You're right. The yaml files of respective nodes are in /etc/<node_name>/elasticsearch.yml, but Elasticsearch reads the configuration from /etc/elasticsearch/elasticsearch.yml.

However, this shouldn't be the case, since all of my nodes have their own config directories, defined in their respective service scripts. Here is an example of elasticsearch_master.service :

[Unit]
Description=Elasticsearch
Documentation=http://www.elastic.co
Wants=network-online.target
After=network-online.target

[Service]
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=CONF_DIR=/etc/master
Environment=DATA_DIR=/var/lib/elasticsearch/master
Environment=LOG_DIR=/var/log/elasticsearch/master
Environment=ES_PATH_CONF=/etc/master
Environment=PID_DIR=/var/run/elasticsearch
EnvironmentFile=-/etc/default/elasticsearch

WorkingDirectory=/usr/share/elasticsearch

User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet

system · August 4, 2019, 3:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Not able to start data node for ES cluster Elasticsearch	1	443	November 14, 2019
Elasticsearch Cluster with 1 master node and 2 data nodes Elasticsearch	2	783	January 25, 2019
Unknown-setting-cluster-initial-master-nodes error in ES 7.6.2 Elasticsearch windows-installer	3	2311	May 23, 2020
Elasticsearch master slave configuration Elasticsearch	2	10276	October 4, 2018
Clustering with Elasticsearch issues Elasticsearch	13	1414	July 5, 2017

Elasticsearch seems to ignore settings in yaml file

Related topics