Thank you Tanguy for looking into this.
Here are the logs when I restart elasticsearch
[root@nl-ams02c-ispdat01 ~]# tail -f /var/log/elasticsearch/*.log &
[1] 37618
[root@nl-ams02c-ispdat01 ~]# systemctl stop elasticsearch
[2015-11-09 12:06:06,174][INFO ][node ] [nl-ams02c-ispdat01] stopping ...
[2015-11-09 12:06:06,315][INFO ][node ] [nl-ams02c-ispdat01] stopped
[2015-11-09 12:06:06,315][INFO ][node ] [nl-ams02c-ispdat01] closing ...
[2015-11-09 12:06:06,330][INFO ][node ] [nl-ams02c-ispdat01] closed
[root@nl-ams02c-ispdat01 ~]# systemctl start elasticsearch
[root@nl-ams02c-ispdat01 ~]# [2015-11-09 12:06:22,039][INFO ][node ] [nl-ams02c-ispdat01] version[2.0.0], pid[37635], build[de54438/2015-10-22T08:09:48Z]
[2015-11-09 12:06:22,039][INFO ][node ] [nl-ams02c-ispdat01] initializing ...
[2015-11-09 12:06:22,301][INFO ][plugins ] [nl-ams02c-ispdat01] loaded [license, marvel], sites [kopf, hq]
[2015-11-09 12:06:22,337][INFO ][env ] [nl-ams02c-ispdat01] using [1] data paths, mounts [[/usr (/dev/mapper/centos-usr)]], net usable_space [1.2tb], net total_space [1.2tb], spins? [possibly], types [xfs]
[2015-11-09 12:06:23,460][INFO ][marvel.agent.settings ] [nl-ams02c-ispdat01] marvel setting [marvel.agent.interval : 1m] updated
[2015-11-09 12:06:24,586][INFO ][node ] [nl-ams02c-ispdat01] initialized
[2015-11-09 12:06:24,586][INFO ][node ] [nl-ams02c-ispdat01] starting ...
[2015-11-09 12:06:24,898][INFO ][transport ] [nl-ams02c-ispdat01] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}
[2015-11-09 12:06:24,908][INFO ][discovery ] [nl-ams02c-ispdat01] IP-spotlight/EBST-cZUQa-jGhnnBkhd0g
[2015-11-09 12:06:27,954][INFO ][cluster.service ] [nl-ams02c-ispdat01] new_master {nl-ams02c-ispdat01}{EBST-cZUQa-jGhnnBkhd0g}{127.0.0.1}{127.0.0.1:9300}{tag=exabgp, group=IP-spotlight}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2015-11-09 12:06:28,026][WARN ][common.network ] [nl-ams02c-ispdat01] publish address: {0.0.0.0} is a wildcard address, falling back to first non-loopback: {fe80::569f:35ff:fe0a:99c0}
[2015-11-09 12:06:28,026][INFO ][http ] [nl-ams02c-ispdat01] publish_address {[fe80::569f:35ff:fe0a:99c0]:9200}, bound_addresses {[::]:9200}
[2015-11-09 12:06:28,028][INFO ][node ] [nl-ams02c-ispdat01] started
[2015-11-09 12:06:28,095][INFO ][license.plugin.core ] [nl-ams02c-ispdat01] license [b568290d-5957-4ab7-bbd4-58fde74bb508] - valid
[2015-11-09 12:06:28,097][ERROR][license.plugin.core ] [nl-ams02c-ispdat01]
#
# License will expire on [Monday, December 07, 2015]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
#
# Commercial plugins operate with reduced functionality on license expiration:
# - marvel
# - The agent will stop collecting cluster and indices metrics
[2015-11-09 12:06:28,120][INFO ][gateway ] [nl-ams02c-ispdat01] recovered [3] indices into cluster_state
[root@nl-ams02c-ispdat01 ~]# kill -SIGTERM 37618
[1]+ Terminated tail -f /var/log/elasticsearch/*.log
i deleted also the ".marvel-es-data" index as well, and re-install marvel also, but in vain.
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana 1 1 3 0 24.3kb 24.3kb
green open <SNIP!> 1 0 819859 0 131.9mb 131.9mb
green open <SNIP!> 1 0 1921373 0 284.4mb 284.4mb
thank you a lot for looking into this. i really appreciate it.
i can provide a team viewer session if necessary in order to troubleshoot the issue.
Nikos
