java.lang.IllegalException: failed to obtain node locks

Elastic Stack Elasticsearch
1

I am setting up ELK stack for the first time on 3 different servers respectively. Elasticsearch + Kibana are good (I think), and now onto logstash I am Parsing Logs following the guide:
https://www.elastic.co/guide/en/logstash/current/advanced-pipeline.html @ the "Testing your pipeline" step. I realize I have no indicies or haven't done anything with the initial config for Elasticsearch:

The date used in the index name is based on UTC, not the timezone where Logstash is running. If the query returns index_not_found_exception, make sure that logstash-$DATE reflects the actual name of the index. To see a list of available indexes, use this query: curl 'localhost:9200/_cat/indices?v'.

So I go over the Elasticsearch server to create an Index for the first (also unsure on the PUT /indexname) API, so I figure I need to run elasticsearch from the bin before I use that command.
elasticsearch/bin$ ./elasticsearch
Error in [Main]:
java.lang.IllegalException: failed to obtain node locks, tried [/var/lib/elasticsearch] with lock id [0]; maybe these locations are note writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?

Now based on the message, the /var location is writeable as I chmodded the errors earlier (I know I know bad practice, but dev). I don't know if I have started a node? Or how to go about increasing max nodes? Shutting down a running node?

Please keep in mind I am a n00b here, but have been making great progress and am just trying to follow the walkthrough. I am trying to add a index on elasticsearch for testing logstash.
Many thanks to anyone who can help!

2

By defaullt, ES will automatically create an index for you if it's not there. You can check /etc/elasticsearch/elasticsearch.yml for all your settings, specifically it can be set
action.auto_create_index: logstash-*
Which will only allow indices with the name logstash-something to be auto-created

This is also where you would set node.max_local_storage_nodes

To check if ES is already running just curl it:
curl -XGET localhost:9200

1 Like
3

I had to actually make it:
action.auto_create_index: ".security*,.monitoring*,..bunch more...,.ml*,logstash-*"
Still having some problems. Perhaps it needs to be .logstash-* and not logstash-*? Checking a few things.

Running curl '<server>:9200/_cat/indices?v' shows no indexes. Is there a way to physically add an index? i.e. how do I use that PUT command - like where - do I put "PUT"?

My thanks again for the help,

4

Actually the '<server>:9200/_cat/indices?v' works on the elasticsearch server, but not logstash server...I appear to no longer be able to just query 9200 either from logstash, ugh, so much troubleshooting lol!

5

Unless your logstash server is running an instance of elasticsearch it won't work for localhost, and would need access to port 9200 on the elasticsearch server.
If it works on the server running elasticsearch it could be a permissions/firewall/iptables problem

Yeah, common index's ES likes to create need to be whitelisted, I have .security,.monitoring*,.watches,.triggered_watches,.watcher-history-*,.ml*, filebeat-*

1 Like
6

Thanks I got it (can see host:9200 from Logstash server). Still the auto index isn't working on logstash server putting in:
curl -XGET 'server:9200/logstash-$DATE/_search?pretty&q=response=200'

Are you saying I also need to install elasticsearch on my logstash server or is it enough that it can just read/write the elasticsearch server?

7

Just access to port 9200 should be enough.

Are you literally putting $DATE or replacing it with some date?

Try curl -XGET server:9200/_cat/indices to see if you have any index's yet.

1 Like
8

THANK YOU SO MUCH. I got it, it turns out my it was logstash-$DATE-00001, so I recommend to everyone to see the indices first.

Thanks again Doc_Kaos!

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.