I have looked through many other threads on this issue, and I cannot seem to find anything that works for me.
I have an Elasticsearch 2.4.0 cluster w/ the appropriate marvel agent configured on all nodes. This has been verified through the GET _cat/plugins
call, where all nodes have these configured:
Data02 license 2.4.0 j
Data02 marvel-agent 2.4.0 j
My license is current and value through Sept 2017 - from GET _license
:
"license": {
"status": "active",
"type": "basic",
"issue_date": "2016-07-29T00:00:00.000Z",
"issue_date_in_millis": 1469750400000,
"expiry_date": "2017-09-13T23:59:59.999Z",
"expiry_date_in_millis": 1505347199999,
"max_nodes": 100
(extraneous & identifying information was removed from the above response)
Marvel DID work for a while. We had some infrastructure issues a few weeks back however and it seems to have stopped working since for some reason.
There are still indexes being maintained:
GET _cat/indices/*marvel*
green open .marvel-2016.09.09 1 1 205521 0 853.9mb 426.9mb
green open .marvel-2016.09.08 1 1 189022 0 810.8mb 405.4mb
green open .marvel-2016.09.07 1 1 1017948 0 2.2gb 1.1gb
green open .marvel-es-1-2017.01.25 1 1 32073 0 23.4mb 7.8mb
green open .marvel-es-1-2017.01.24 1 1 55493 0 28.7mb 14.4mb
green open .marvel-es-1-2017.01.19 1 1 59577 0 26.7mb 13.3mb
green open .marvel-2016.09.10 1 1 36158 0 139.3mb 69.6mb
green open .marvel-es-1-2017.01.23 1 1 56742 0 28.5mb 14.3mb
green open .marvel-es-data-1 1 1 19 7486 681kb 340.5kb
green open .marvel-es-1-2017.01.22 1 1 59341 0 24.9mb 12.4mb
green open .marvel-es-1-2017.01.21 1 1 59518 0 24.5mb 12.3mb
green open .marvel-es-1-2017.01.20 1 1 59576 0 26.5mb 13.3mb
green open .marvel-kibana 1 1 1 0 6.5kb 3.2kb
We did upgrade from 1.7 -> 2.4 last september so some older indices are still lingering about, and we configured marvel to only roll 7 days of logs.
I added an alias .marvel-es-data which references .marvel-es-data-1
per one suggestion I found, but to no avail.
Looking at today's index (.marvel-es-1-2017.01.25
) to see which nodes are reporting, I did note that two nodes are missing: one of the Data-only nodes and one of the Master nodes (the one that happens to be the current master). I don't know that this is the cause of the issue, but I'm going to look to restart both services off-hours. Maybe it is?
There are no errors reported in any logs that seem relevant, and the Sense UI via Kibana works fine. The ELASTICSEARCH_URL parameter looks good as well (Kibana & its plugins are in a docker container so I've been rebuilding that regularly trying to get it to work again).
Should the .marvel-es-1-{yyyy.mm.dd}
nodes be part of some alias perhaps? I'm really baffled as to what else could be wrong.