Failed to obtain node lock

Zoran_Jeremic · July 2, 2013, 12:06am

Hi,

I have the following problem if I'm using local node. It works find with
elasticsearch server, but if I switch to the local node, then application
creates 50 nodes, and the following error is fired:

Exception in thread "Thread-500" org.elasticsearch.
ElasticSearchIllegalStateException: Failed to obtain node lock, is the
following location writable?: [/home/zoran/elasticsearch/elasticsearch]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:123
)
at org.elasticsearch.node.internal.InternalNode.(InternalNode.java
:134)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(
ElasticSearchFactory.java:62)
at org.prosolo.services.indexing.ElasticSearchFactory.getClient(
ElasticSearchFactory.java:22)
at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.
indexNode(AbstractBaseEntityESServiceImpl.java:57)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.
saveResourceNode(NodeEntityESServiceImpl.java:50)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.
saveNodeToES(NodeEntityESServiceImpl.java:30)
at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(
NodeChangeObserver.java:53)
at java.lang.Thread.run(Thread.java:724)

I know there is the limit on the OS to maximum number of opened files, so I
changed /etc/security/limits.conf

zoran soft nofile 15000
zoran hard nofile 15000

However, it doesn't work, so I suppose that I should make elasticsearch
aware of this change in some way.

Source code creating local client is like this:

private static Client getLocalClient(){
ElasticSearchConfig elasticSearchConfig=Settings.getInstance().
config.elasticSearch;
String dataDirectory = elasticSearchConfig.homePath;

    ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.

settingsBuilder()
// .put("http.enabled", "false")
.put("cluster.name", elasticSearchConfig.clusterName)
.put("path.data", dataDirectory);

   Node node = nodeBuilder()
            .local(true)
            .settings(elasticsearchSettings.build())
            .node();
       Client client = node.client();
    return client;

}

I'm sure that this location is writeable as it index first 50 documents
without a problem.

Could you please tell me what could be the problem?

Best,
Zoran

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Zoran_Jeremic · July 2, 2013, 5:40am

Hi,

I just solved this problem. It was related to the mistake in the code as I
was initiated new client on each request instead of using single client for
all requests in the application.

Zoran

On Monday, 1 July 2013 17:06:12 UTC-7, Zoran Jeremic wrote:

Hi,

I have the following problem if I'm using local node. It works find with
elasticsearch server, but if I switch to the local node, then application
creates 50 nodes, and the following error is fired:

Exception in thread "Thread-500" org.elasticsearch.
ElasticSearchIllegalStateException: Failed to obtain node lock, is the
following location writable?: [/home/zoran/elasticsearch/elasticsearch]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:
123)
at org.elasticsearch.node.internal.InternalNode.(InternalNode.
java:134)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(
ElasticSearchFactory.java:62)
at org.prosolo.services.indexing.ElasticSearchFactory.getClient(
ElasticSearchFactory.java:22)
at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.
indexNode(AbstractBaseEntityESServiceImpl.java:57)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.
saveResourceNode(NodeEntityESServiceImpl.java:50)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.
saveNodeToES(NodeEntityESServiceImpl.java:30)
at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(
NodeChangeObserver.java:53)
at java.lang.Thread.run(Thread.java:724)

I know there is the limit on the OS to maximum number of opened files, so
I changed /etc/security/limits.conf

zoran soft nofile 15000
zoran hard nofile 15000

However, it doesn't work, so I suppose that I should make elasticsearch
aware of this change in some way.

Source code creating local client is like this:

private static Client getLocalClient(){
ElasticSearchConfig elasticSearchConfig=Settings.getInstance().
config.elasticSearch;
String dataDirectory = elasticSearchConfig.homePath;
    ImmutableSettings.Builder elasticsearchSettings = 
ImmutableSettings.settingsBuilder()
// .put("http.enabled", "false")
.put("cluster.name", elasticSearchConfig.clusterName)
.put("path.data", dataDirectory);
   Node node = nodeBuilder()
            .local(true)
            .settings(elasticsearchSettings.build())
            .node();
       Client client = node.client();
    return client;

}
I'm sure that this location is writeable as it index first 50 documents
without a problem.

Could you please tell me what could be the problem?

Best,
Zoran

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vyeluri5 · September 21, 2014, 9:42am

Hi,

I have a similar problem. Ive defined complete write permission for my
folder that Ive defined in data.path: /var/lib/elasticsearch/elasticsearch
I still get the error Failed to obtain node lock, is the following location
writable?: [/var/lib/elasticsearch/elasticsearch]

Could you tell me where I`m doing wrong??

On Tuesday, July 2, 2013 10:06:12 AM UTC+10, Zoran Jeremic wrote:

Hi,

I have the following problem if I'm using local node. It works find with
elasticsearch server, but if I switch to the local node, then application
creates 50 nodes, and the following error is fired:

Exception in thread "Thread-500" org.elasticsearch.
ElasticSearchIllegalStateException: Failed to obtain node lock, is the
following location writable?: [/home/zoran/elasticsearch/elasticsearch]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:
123)
at org.elasticsearch.node.internal.InternalNode.(InternalNode.
java:134)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(
ElasticSearchFactory.java:62)
at org.prosolo.services.indexing.ElasticSearchFactory.getClient(
ElasticSearchFactory.java:22)
at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.
indexNode(AbstractBaseEntityESServiceImpl.java:57)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.
saveResourceNode(NodeEntityESServiceImpl.java:50)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.
saveNodeToES(NodeEntityESServiceImpl.java:30)
at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(
NodeChangeObserver.java:53)
at java.lang.Thread.run(Thread.java:724)

I know there is the limit on the OS to maximum number of opened files, so
I changed /etc/security/limits.conf

zoran soft nofile 15000
zoran hard nofile 15000

However, it doesn't work, so I suppose that I should make elasticsearch
aware of this change in some way.

Source code creating local client is like this:

private static Client getLocalClient(){
ElasticSearchConfig elasticSearchConfig=Settings.getInstance().
config.elasticSearch;
String dataDirectory = elasticSearchConfig.homePath;
    ImmutableSettings.Builder elasticsearchSettings = 
ImmutableSettings.settingsBuilder()
// .put("http.enabled", "false")
.put("cluster.name", elasticSearchConfig.clusterName)
.put("path.data", dataDirectory);
   Node node = nodeBuilder()
            .local(true)
            .settings(elasticsearchSettings.build())
            .node();
       Client client = node.client();
    return client;

}
I'm sure that this location is writeable as it index first 50 documents
without a problem.

Could you please tell me what could be the problem?

Best,
Zoran

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · September 21, 2014, 10:34pm

What are the ownership and permissions on the directory?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 21 September 2014 19:42, vyeluri vyeluri5@gmail.com wrote:

Hi,

I have a similar problem. Ive defined complete write permission for my
folder that Ive defined in data.path: /var/lib/elasticsearch/elasticsearch
I still get the error Failed to obtain node lock, is the following
location writable?: [/var/lib/elasticsearch/elasticsearch]

Could you tell me where I`m doing wrong??

On Tuesday, July 2, 2013 10:06:12 AM UTC+10, Zoran Jeremic wrote:
Hi,

I have the following problem if I'm using local node. It works find with
elasticsearch server, but if I switch to the local node, then application
creates 50 nodes, and the following error is fired:

Exception in thread "Thread-500" org.elasticsearch.ElasticSearc
hIllegalStateException: Failed to obtain node lock, is the following
location writable?: [/home/zoran/elasticsearch/elasticsearch]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:
123)
at org.elasticsearch.node.internal.InternalNode.(InternalNode.
java:134)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:159)
at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:166)
at org.prosolo.services.indexing.ElasticSearchFactory.getLocalClient(
ElasticSearchFactory.java:62)
at org.prosolo.services.indexing.ElasticSearchFactory.getClient(
ElasticSearchFactory.java:22)
at org.prosolo.services.indexing.AbstractBaseEntityESServiceImpl.
indexNode(AbstractBaseEntityESServiceImpl.java:57)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.s
aveResourceNode(NodeEntityESServiceImpl.java:50)
at org.prosolo.services.indexing.impl.NodeEntityESServiceImpl.s
aveNodeToES(NodeEntityESServiceImpl.java:30)
at org.prosolo.services.indexing.impl.NodeChangeObserver$1.run(
NodeChangeObserver.java:53)
at java.lang.Thread.run(Thread.java:724)

I know there is the limit on the OS to maximum number of opened files, so
I changed /etc/security/limits.conf

zoran soft nofile 15000
zoran hard nofile 15000

However, it doesn't work, so I suppose that I should make elasticsearch
aware of this change in some way.

Source code creating local client is like this:

private static Client getLocalClient(){
ElasticSearchConfig elasticSearchConfig=Settings.getInstance().
config.elasticSearch;
String dataDirectory = elasticSearchConfig.homePath;
    ImmutableSettings.Builder elasticsearchSettings =
ImmutableSettings.settingsBuilder()
// .put("http.enabled", "false")
.put("cluster.name", elasticSearchConfig.clusterName)
.put("path.data", dataDirectory);
   Node node = nodeBuilder()
            .local(true)
            .settings(elasticsearchSettings.build())
            .node();
       Client client = node.client();
    return client;

}
I'm sure that this location is writeable as it index first 50 documents
without a problem.

Could you please tell me what could be the problem?

Best,
Zoran

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03e5c589-d8d9-4048-9465-1b99e24da8af%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b7T3q7ZetxUyRt3sw4VfCpVWx51rEmduPOU6RGQDVWmQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Rajnikant · July 29, 2016, 8:08am

Hi @Zoran_Jeremic,
How can i use one client for all request in client?
can you give some brief explanation about about my query?
My application is start successfully but after few request i got this type of error ---

"java.lang.IllegalStateException: Failed to obtain node lock, is the following location writable?"

Zoran_Jeremic · July 29, 2016, 4:46pm

Hi @Rajnikant,

It depends on the design of your application. In my case, I have a Spring service ElasticsearchFactory that keeps instance of the client. This service is called in each request. Part of the code is given bellow, but you can use singleton or some other approach to make sure you are not instantiating new client each time.

 @Service("org.prosolo.services.indexing.ElasticSearchFactory")
 public class ElasticSearchFactory {

private static Client client;
private static Logger logger = Logger.getLogger(ElasticSearchFactory.class);

public static Client getClient() {
  if (client == null) {
     if (CommonSettings.getInstance().config.elasticSearch.type.equals("local")) {
        client = getLocalClient();
     } else if (CommonSettings.getInstance().config.elasticSearch.type.equals("server")){
        client = getESClient();
        }else if (CommonSettings.getInstance().config.elasticSearch.type.equals("cloud-aws")){
        client = getAWSClient();
     }
   }
  
  return client;
}

private static Client getESClient() {
  ElasticSearchConfig elasticSearchConfig = CommonSettings.getInstance().config.elasticSearch;
  try {
     Settings settings = Settings.settingsBuilder()
           .put("cluster.name", elasticSearchConfig.clusterName).build();
     ArrayList<ElasticSearchHost> esHosts = elasticSearchConfig.esHostsConfig.esHosts;
  client=TransportClient.builder().settings(settings).build();
     for (ElasticSearchHost host : esHosts) {
        ((TransportClient) client).addTransportAddress(new InetSocketTransportAddress(new InetSocketAddress(host.host, host.port)));
        
     }
     ClusterHealthResponse clusterHealth = client.admin().cluster().health(clusterHealthRequest().waitForGreenStatus()).actionGet();
  } catch (NoNodeAvailableException ex) {
     logger.error("ElasticSearch node is not available. " + ex);
     //throw new IndexingServiceNotAvailable("ElasticSearch node is not available. " + ex);
  } catch (Exception ex) {
     logger.error("Exception for cluster:" + elasticSearchConfig.clusterName, ex);
     return null;
  }
  return client;
}

Hope this helps.
Zoran

Rajnikant · August 1, 2016, 7:54am

thank you so much.. @Zoran_Jeremic.