I recently re-installed elastic having upgraded from our previous 1.x version to 2.1.1.
I didn't setup the initial install so not 100% on the configuration. But currently my cluster has become red.
Looking at the cluster and indices I've found the following:
elastic@elk:/apps/elasticsearch-2.1.1/logs$ curl 'localhost:9200/_cat/nodes?v' host ip heap.percent ram.percent load node.role master name 127.0.0.1 127.0.0.1 5 28 0.09 d * elasticsearch elastic@elk:/apps/elasticsearch-2.1.1/logs$ curl 'localhost:9200/_cat/indices?v' health status index pri rep docs.count docs.deleted store.size pri.store.size yellow open logstash-2016.01.13 10 1 732832 0 145.3mb 145.3mb yellow open logstash-2016.01.14 10 1 1758432 0 320.8mb 320.8mb red open logstash-2016.01.15 10 1 381676 0 147.4mb 147.4mb yellow open .kibana 1 1 3 0 10.1kb 10.1kb
There's also two elastic processes running, the top is the current process and seems to be erroring. The second process is a zombie.
elastic@elk:/apps/elasticsearch-2.1.1/logs$ ps aux | grep elasticsearch elastic 2317 36.7 3.4 5940116 420564 pts/1 Sl 10:29 2:31 /usr/bin/java -Xms256m -Xmx1g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true -Des.path.home=/apps/elasticsearch-2.1.1 -cp /apps/elasticsearch-2.1.1/lib/elasticsearch-2.1.1.jar:/apps/elasticsearch-2.1.1/lib/* org.elasticsearch.bootstrap.Elasticsearch start -d -p PID elastic 2706 0.0 0.0 8216 2228 pts/1 R+ 10:36 0:00 grep --color=auto elasticsearch
Looking at the logs for startup we have a few warnings:
[2016-01-15 10:29:43,609][WARN ][bootstrap ] Unable to lock JVM Memory: error=12,reason=Cannot allocate memory [2016-01-15 10:29:43,609][WARN ][bootstrap ] This can result in part of the JVM being swapped out. [2016-01-15 10:29:43,609][WARN ][bootstrap ] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536 [2016-01-15 10:29:43,609][WARN ][bootstrap ] These can be adjusted by modifying /etc/security/limits.conf, for example: # allow user 'elastic' mlockall elastic soft memlock unlimited elastic hard memlock unlimited [2016-01-15 10:29:43,609][WARN ][bootstrap ] If you are logged in interactively, you will have to re-login for the new limits to take effect.
This is an ELK stack. Does anyone know how I can get this to green?