I started a new installation ELK server(standalone ELK installed on the sameserver) and we noted that kibana doesn't show any of data since 07.06, so i checked the logs and
ls -altr /opt/elkdata/elasticsearch/elasticsearch/nodes/0/indices/
Jul 3 09:42 logstash-2015.07.03
Jul 3 09:54 .kibana
Jul 4 02:00 logstash-2015.07.04
Jul 5 02:00 logstash-2015.07.05
Jul 6 02:00 .
Jul 6 02:00 logstash-2015.07.06
Jul 7 13:41 ..
but
ls -altr /opt/elkdata/elasticsearch/logs/
0 Jun 16 14:47 elasticsearch_index_indexing_slowlog.log
0 Jun 16 14:47 elasticsearch_index_search_slowlog.log
4096 Jun 17 10:57 ..
3461 Jun 17 10:58 elasticsearch.log.2015-06-17
10311 Jul 3 15:17 elasticsearch.log.2015-07-03
5731 Jul 4 14:36 elasticsearch.log.2015-07-04
6689 Jul 5 21:27 elasticsearch.log.2015-07-05
21343 Jul 6 11:54 elasticsearch.log.2015-07-06
4096 Jul 7 13:41 .
2901 Jul 7 13:42 elasticsearch.log
last log entry in elasticsearch.log:
[2015-07-07 13:41:06,533][INFO ][cluster.service ] [logstash] removed {[logstash-hostname.org-16216-13418][M19k1C2cSDezf5mu1IxvcQ][hostname.org][inet[/192.168.1.2:9301]]{data=false, client=true},}, reason: zen-disco-node_failed([logstash-hostname.org-16216-13418][M19k1C2cSDezf5mu1IxvcQ][hostname.org][inet[/192.168.1.2:9301]]{data=false, client=true}), reason transport disconnected
[2015-07-07 13:41:16,921][INFO ][node ] [logstash] stopping ...
[2015-07-07 13:41:17,334][INFO ][node ] [logstash] stopped
[2015-07-07 13:41:17,335][INFO ][node ] [logstash] closing ...
[2015-07-07 13:41:17,357][INFO ][node ] [logstash] closed
[2015-07-07 13:41:49,954][INFO ][node ] [logstash] version[1.6.0], pid[9936], build[cdd3ac4/2015-06-09T13:36:34Z]
[2015-07-07 13:41:49,955][INFO ][node ] [logstash] initializing ...
[2015-07-07 13:41:49,964][INFO ][plugins ] [logstash] loaded [], sites []
[2015-07-07 13:41:50,061][INFO ][env ] [logstash] using [1] data paths, mounts [[/opt/elkdata (/dev/mapper/ccblst01vg-elkdata)]], net usable_space [814gb], net total_space [944.8gb], types [ext4]
[2015-07-07 13:41:53,864][INFO ][node ] [logstash] initialized
[2015-07-07 13:41:53,865][INFO ][node ] [logstash] starting ...
[2015-07-07 13:41:53,941][INFO ][transport ] [logstash] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/192.168.1.2:9300]}
[2015-07-07 13:41:53,955][INFO ][discovery ] [logstash] elasticsearch/cLwoLX7sQIyi3zcpnkmiGg
[2015-07-07 13:41:57,748][INFO ][cluster.service ] [logstash] new_master [logstash][cLwoLX7sQIyi3zcpnkmiGg][hostname.org][inet[/192.168.1.2:9300]]{master=true}, reason: zen-disco-join (elected_as_master)
[2015-07-07 13:41:57,937][INFO ][http ] [logstash] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/192.168.1.2:9200]}
[2015-07-07 13:41:57,938][INFO ][node ] [logstash] started
[2015-07-07 13:41:58,003][INFO ][gateway ] [logstash] recovered [5] indices into cluster_state
[2015-07-07 13:42:22,620][INFO ][cluster.service ] [logstash] added {[logstash-hostname.org-10009-13434][_HLcxrLuRViCZq_lvscgvQ][hostname.org][inet[/192.168.1.2:9301]]{data=false, client=true},}, reason: zen-disco-receive(join from node[[logstash-hostname.org-10009-13434][_HLcxrLuRViCZq_lvscgvQ][hostname.org][inet[/192.168.1.2:9301]]{data=false, client=true}])
LS:
last entry of logstash.log
{:timestamp=>"2015-07-07T13:42:15.728000+0200", :message=>"Adding pattern",
kibana last entry:
July 6th 2015, 11:54:50.442 message: 2015-07-06T10:54:01.977;
the box is 32Gb 4CPU 1TB and here is the ls and es start param
ES:
/usr/bin/java -Xms16g -Xmx16g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Delasticsearch -Des.pidfile=-Des.index.store.type=memory -Des.path.home=/opt/elasticsearch -cp :/opt/elasticsearch/lib/elasticsearch-1.6.0.jar:/opt/elasticsearch/lib/:/opt/elasticsearch/lib/sigar/ -Des.node.name=ccb_logstash_tud -Des.config=/opt/elasticsearch/config/elasticsearch.yml -Des.path.home=/opt/elasticsearch -Des.path.logs=/opt/elkdata/elasticsearch/logs -Des.path.data=/opt/elkdata/elasticsearch -Des.path.work=//tmp org.elasticsearch.bootstrap.Elasticsearch
LS:
/usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xmx1G -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent --verbose -f /opt/logstash/conf --log /opt/elkdata/logstash/logstash.log