# Failed to obtain node lock

**URL:** <https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608>\
**Category:** Elasticsearch\
**Created:** [July 2, 2013, 12:06am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608 "2013-07-02T00:06:12Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\
**Post date:** [July 2, 2013, 12:06am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/1 "2013-07-02T00:06:12Z")

</div>

Hi,

I have the following problem if I'm using local node. It works find with  
elasticsearch server, but if I switch to the local node, then application  
creates 50 nodes, and the following error is fired:

Exception in thread "Thread-500" org.elasticsearch.  
ElasticSearchIllegalStateException: Failed to obtain node lock, is the  
following location writable?: [/home/zoran/elasticsearch/elasticsearch]  
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:123  
)  
at org.elasticsearch.node.internal.InternalNode.(InternalNode.java  
:134)  
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)  
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)  
at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(  
ElasticSearchFactory.java:62)  
at org.prosolo.services.indexing.ElasticSearchFactory.getClient(  
ElasticSearchFactory.java:22)  
at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.  
indexNode(AbstractBaseEntityESServiceImpl.java:57)  
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.  
saveResourceNode(NodeEntityESServiceImpl.java:50)  
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.  
saveNodeToES(NodeEntityESServiceImpl.java:30)  
at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(  
NodeChangeObserver.java:53)  
at java.lang.Thread.run(Thread.java:724)

I know there is the limit on the OS to maximum number of opened files, so I  
changed /etc/security/limits.conf

zoran soft nofile 15000  
zoran hard nofile 15000

However, it doesn't work, so I suppose that I should make elasticsearch  
aware of this change in some way.

Source code creating local client is like this:

private static Client getLocalClient(){  
ElasticSearchConfig elasticSearchConfig=Settings.getInstance().  
config.elasticSearch;  
String dataDirectory = elasticSearchConfig.homePath;

```
    ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.

```

settingsBuilder()  
// .put("http.enabled", "false")  
.put("cluster.name", elasticSearchConfig.clusterName)  
.put("path.data", dataDirectory);

```
   Node node = nodeBuilder()
            .local(true)
            .settings(elasticsearchSettings.build())
            .node();
       Client client = node.client();
    return client;

}

```

I'm sure that this location is writeable as it index first 50 documents  
without a problem.

Could you please tell me what could be the problem?

Best,  
Zoran

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\
**Post date:** [July 2, 2013, 5:40am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/2 "2013-07-02T05:40:50Z")

</div>

Hi,

I just solved this problem. It was related to the mistake in the code as I  
was initiated new client on each request instead of using single client for  
all requests in the application.

Zoran

On Monday, 1 July 2013 17:06:12 UTC-7, Zoran Jeremic wrote:

> Hi,
> 
> I have the following problem if I'm using local node. It works find with  
> elasticsearch server, but if I switch to the local node, then application  
> creates 50 nodes, and the following error is fired:
> 
> Exception in thread "Thread-500" org.elasticsearch.  
> ElasticSearchIllegalStateException: Failed to obtain node lock, is the  
> following location writable?: [/home/zoran/elasticsearch/elasticsearch]  
> at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:  
> 123)  
> at org.elasticsearch.node.internal.InternalNode.(InternalNode.  
> java:134)  
> at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)  
> at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)  
> at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(  
> ElasticSearchFactory.java:62)  
> at org.prosolo.services.indexing.ElasticSearchFactory.getClient(  
> ElasticSearchFactory.java:22)  
> at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.  
> indexNode(AbstractBaseEntityESServiceImpl.java:57)  
> at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.  
> saveResourceNode(NodeEntityESServiceImpl.java:50)  
> at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.  
> saveNodeToES(NodeEntityESServiceImpl.java:30)  
> at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(  
> NodeChangeObserver.java:53)  
> at java.lang.Thread.run(Thread.java:724)
> 
> I know there is the limit on the OS to maximum number of opened files, so  
> I changed /etc/security/limits.conf
> 
> zoran soft nofile 15000  
> zoran hard nofile 15000
> 
> However, it doesn't work, so I suppose that I should make elasticsearch  
> aware of this change in some way.
> 
> Source code creating local client is like this:
> 
> private static Client getLocalClient(){  
> ElasticSearchConfig elasticSearchConfig=Settings.getInstance().  
> config.elasticSearch;  
> String dataDirectory = elasticSearchConfig.homePath;
> 
> ```
> ImmutableSettings.Builder elasticsearchSettings = 
> 
> ```
> 
> ImmutableSettings.settingsBuilder()  
> // .put("http.enabled", "false")  
> .put("cluster.name", elasticSearchConfig.clusterName)  
> .put("path.data", dataDirectory);
> 
> ```
> Node node = nodeBuilder()
> .local(true)
> .settings(elasticsearchSettings.build())
> .node();
> Client client = node.client();
> return client;
> 
> }
> 
> ```
> 
> I'm sure that this location is writeable as it index first 50 documents  
> without a problem.
> 
> Could you please tell me what could be the problem?
> 
> Best,  
> Zoran

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![vyeluri5](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vyeluri5/32/1269_2.png) [@vyeluri5](https://discuss.elastic.co/u/vyeluri5)\
**Post date:** [September 21, 2014, 9:42am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/3 "2014-09-21T09:42:53Z")

</div>

Hi,

I have a similar problem. Ive defined complete write permission for my  
folder that Ive defined in data.path: /var/lib/elasticsearch/elasticsearch  
I still get the error Failed to obtain node lock, is the following location  
writable?: [/var/lib/elasticsearch/elasticsearch]

Could you tell me where I`m doing wrong??

On Tuesday, July 2, 2013 10:06:12 AM UTC+10, Zoran Jeremic wrote:

> Hi,
> 
> I have the following problem if I'm using local node. It works find with  
> elasticsearch server, but if I switch to the local node, then application  
> creates 50 nodes, and the following error is fired:
> 
> Exception in thread "Thread-500" org.elasticsearch.  
> ElasticSearchIllegalStateException: Failed to obtain node lock, is the  
> following location writable?: [/home/zoran/elasticsearch/elasticsearch]  
> at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:  
> 123)  
> at org.elasticsearch.node.internal.InternalNode.(InternalNode.  
> java:134)  
> at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)  
> at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)  
> at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(  
> ElasticSearchFactory.java:62)  
> at org.prosolo.services.indexing.ElasticSearchFactory.getClient(  
> ElasticSearchFactory.java:22)  
> at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.  
> indexNode(AbstractBaseEntityESServiceImpl.java:57)  
> at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.  
> saveResourceNode(NodeEntityESServiceImpl.java:50)  
> at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.  
> saveNodeToES(NodeEntityESServiceImpl.java:30)  
> at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(  
> NodeChangeObserver.java:53)  
> at java.lang.Thread.run(Thread.java:724)
> 
> I know there is the limit on the OS to maximum number of opened files, so  
> I changed /etc/security/limits.conf
> 
> zoran soft nofile 15000  
> zoran hard nofile 15000
> 
> However, it doesn't work, so I suppose that I should make elasticsearch  
> aware of this change in some way.
> 
> Source code creating local client is like this:
> 
> private static Client getLocalClient(){  
> ElasticSearchConfig elasticSearchConfig=Settings.getInstance().  
> config.elasticSearch;  
> String dataDirectory = elasticSearchConfig.homePath;
> 
> ```
> ImmutableSettings.Builder elasticsearchSettings = 
> 
> ```
> 
> ImmutableSettings.settingsBuilder()  
> // .put("http.enabled", "false")  
> .put("cluster.name", elasticSearchConfig.clusterName)  
> .put("path.data", dataDirectory);
> 
> ```
> Node node = nodeBuilder()
> .local(true)
> .settings(elasticsearchSettings.build())
> .node();
> Client client = node.client();
> return client;
> 
> }
> 
> ```
> 
> I'm sure that this location is writeable as it index first 50 documents  
> without a problem.
> 
> Could you please tell me what could be the problem?
> 
> Best,  
> Zoran

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 21, 2014, 10:34pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/4 "2014-09-21T22:34:32Z")

</div>

What are the ownership and permissions on the directory?

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 21 September 2014 19:42, vyeluri [vyeluri5@gmail.com](mailto:vyeluri5@gmail.com) wrote:

> Hi,
> 
> I have a similar problem. Ive defined complete write permission for my  
> folder that Ive defined in data.path: /var/lib/elasticsearch/elasticsearch  
> I still get the error Failed to obtain node lock, is the following  
> location writable?: [/var/lib/elasticsearch/elasticsearch]
> 
> Could you tell me where I`m doing wrong??
> 
> On Tuesday, July 2, 2013 10:06:12 AM UTC+10, Zoran Jeremic wrote:
> 
> > Hi,
> > 
> > I have the following problem if I'm using local node. It works find with  
> > elasticsearch server, but if I switch to the local node, then application  
> > creates 50 nodes, and the following error is fired:
> > 
> > Exception in thread "Thread-500" org.elasticsearch.ElasticSearc  
> > hIllegalStateException: Failed to obtain node lock, is the following  
> > location writable?: [/home/zoran/elasticsearch/elasticsearch]  
> > at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:  
> > 123)  
> > at org.elasticsearch.node.internal.InternalNode.(InternalNode.  
> > java:134)  
> > at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)  
> > at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)  
> > at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(  
> > ElasticSearchFactory.java:62)  
> > at org.prosolo.services.indexing.ElasticSearchFactory.getClient(  
> > ElasticSearchFactory.java:22)  
> > at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.  
> > indexNode(AbstractBaseEntityESServiceImpl.java:57)  
> > at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.s  
> > aveResourceNode(NodeEntityESServiceImpl.java:50)  
> > at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.s  
> > aveNodeToES(NodeEntityESServiceImpl.java:30)  
> > at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(  
> > NodeChangeObserver.java:53)  
> > at java.lang.Thread.run(Thread.java:724)
> > 
> > I know there is the limit on the OS to maximum number of opened files, so  
> > I changed /etc/security/limits.conf
> > 
> > zoran soft nofile 15000  
> > zoran hard nofile 15000
> > 
> > However, it doesn't work, so I suppose that I should make elasticsearch  
> > aware of this change in some way.
> > 
> > Source code creating local client is like this:
> > 
> > private static Client getLocalClient(){  
> > ElasticSearchConfig elasticSearchConfig=Settings.getInstance().  
> > config.elasticSearch;  
> > String dataDirectory = elasticSearchConfig.homePath;
> > 
> > ```
> > ImmutableSettings.Builder elasticsearchSettings =
> > 
> > ```
> > 
> > ImmutableSettings.settingsBuilder()  
> > // .put("http.enabled", "false")  
> > .put("cluster.name", elasticSearchConfig.clusterName)  
> > .put("path.data", dataDirectory);
> > 
> > ```
> > Node node = nodeBuilder()
> > .local(true)
> > .settings(elasticsearchSettings.build())
> > .node();
> > Client client = node.client();
> > return client;
> > 
> > }
> > 
> > ```
> > 
> > I'm sure that this location is writeable as it index first 50 documents  
> > without a problem.
> > 
> > Could you please tell me what could be the problem?
> > 
> > Best,  
> > Zoran
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624b7T3q7ZetxUyRt3sw4VfCpVWx51rEmduPOU6RGQDVWmQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624b7T3q7ZetxUyRt3sw4VfCpVWx51rEmduPOU6RGQDVWmQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Rajnikant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajnikant/32/6490_2.png) [@Rajnikant](https://discuss.elastic.co/u/Rajnikant)\
**Post date:** [July 29, 2016, 8:08am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/5 "2016-07-29T08:08:04Z")

</div>

Hi @Zoran_Jeremic,  
How can i use one client for all request in client?  
can you give some brief explanation about about my query?  
My application is start successfully but after few request i got this type of error ---

"java.lang.IllegalStateException: Failed to obtain node lock, is the following location writable?"

---

<div class="post-metadata">

**Author:** ![Zoran\_Jeremic](https://avatars.discourse-cdn.com/v4/letter/z/7ba0ec/32.png) [@Zoran\_Jeremic](https://discuss.elastic.co/u/Zoran_Jeremic)\
**Post date:** [July 29, 2016, 4:46pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/6 "2016-07-29T16:46:37Z")

</div>

Hi @Rajnikant,

It depends on the design of your application. In my case, I have a Spring service ElasticsearchFactory that keeps instance of the client. This service is called in each request. Part of the code is given bellow, but you can use singleton or some other approach to make sure you are not instantiating new client each time.

> ```
> @Service("org.prosolo.services.indexing.ElasticSearchFactory")
> public class ElasticSearchFactory {
> 
> private static Client client;
> private static Logger logger = Logger.getLogger(ElasticSearchFactory.class);
> 
> ```

> ```
> public static Client getClient() {
> if (client == null) {
> if (CommonSettings.getInstance().config.elasticSearch.type.equals("local")) {
> client = getLocalClient();
> } else if (CommonSettings.getInstance().config.elasticSearch.type.equals("server")){
> client = getESClient();
> }else if (CommonSettings.getInstance().config.elasticSearch.type.equals("cloud-aws")){
> client = getAWSClient();
> }
> }
>   
> return client;
> }
> 
> ```

> ```
> private static Client getESClient() {
> ElasticSearchConfig elasticSearchConfig = CommonSettings.getInstance().config.elasticSearch;
> try {
> Settings settings = Settings.settingsBuilder()
> .put("cluster.name", elasticSearchConfig.clusterName).build();
> ArrayList<ElasticSearchHost> esHosts = elasticSearchConfig.esHostsConfig.esHosts;
> client=TransportClient.builder().settings(settings).build();
> for (ElasticSearchHost host : esHosts) {
> ((TransportClient) client).addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress(host.host, host.port)));
>         
> }
> ClusterHealthResponse clusterHealth = client.admin().cluster().health(clusterHealthRequest().waitForGreenStatus()).actionGet();
> } catch (NoNodeAvailableException ex) {
> logger.error("ElasticSearch node is not available. " + ex);
> //throw new IndexingServiceNotAvailable("ElasticSearch node is not available. " + ex);
> } catch (Exception ex) {
> logger.error("Exception for cluster:" + elasticSearchConfig.clusterName, ex);
> return null;
> }
> return client;
> }
> 
> ```

Hope this helps.  
Zoran

---

<div class="post-metadata">

**Author:** ![Rajnikant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajnikant/32/6490_2.png) [@Rajnikant](https://discuss.elastic.co/u/Rajnikant)\
**Post date:** [August 1, 2016, 7:54am UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/7 "2016-08-01T07:54:04Z")

</div>

thank you so much.. @Zoran_Jeremic.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:31pm UTC](https://discuss.elastic.co/t/failed-to-obtain-node-lock/12608/8 "2017-07-05T22:31:19Z")

</div>