Waiting for Marvel Data

Elastic Stack Kibana
1

Hey guys,
We have been running ES + Kibana + Marvel for around 6 months, but today we decided to change the nodes since we required more disk space. After we deployed everything on the new machines, no data is showing up in Marvel.

Elasticsearch version: 2.3.1
Kibana verison: 4.4.1

The index for marvel exists:

    green open b-2016-5-18_16       2 1        0    0   636b   318b
    green open b-2016-5-18_17       2 1        0    0   636b   318b
    green open n-2016-5-18_20     2 1        0    0   520b   260b
    green open .marvel-es-data-1       1 1        7   65 41.2kb 19.7kb
    green open .kibana                 1 1        5    0 79.4kb 39.7kb
    green open b-2016-5-18_18       2 1   587316  137    1gb  526mb
    green open b-2016-5-18_19       2 1  1977878 8281  3.8gb  1.8gb
    green open .marvel-es-1-2016.05.18 1 1     8322  152  7.6mb  3.8mb
    green open b-2016-5-18_20       2 1        0    0   520b   260b
    green open n-2016-5-18_18     2 1 12566512    0 14.2gb    7gb
    green open n-2016-5-18_19     2 1 43850143    0 57.7gb 29.5gb

Plugins (/_cat/plugins):

    query  license      2.3.1 j
    query  marvel-agent 2.3.1 j
    slave  license      2.3.1 j
    slave  marvel-agent 2.3.1 j
    master license      2.3.1 j
    master marvel-agent 2.3.1 j

In the logs I do see some issues:
[2016-05-18 19:19:14,588][ERROR][marvel.agent.collector.node] [master] collector [node-stats-collector] - failed collecting data
java.lang.ArrayIndexOutOfBoundsException: 0
at org.elasticsearch.action.support.nodes.BaseNodesResponse.getAt(BaseNodesResponse.java:72)
at org.elasticsearch.marvel.agent.collector.node.NodeStatsCollector.doCollect(NodeStatsCollector.java:88)
at org.elasticsearch.marvel.agent.collector.AbstractCollector.collect(AbstractCollector.java:99)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:187)

I'm able to see the data in Kibana under .marvel-es-* index. Anyone can point me in the right direction as to why this issue is happening?

2

The problem is that you are running ES 2.3 with Kibana 4.4.

Instead, you should be running ES 2.3 with Kibana 4.5. Fortunately this annoying version mismatch will take care of itself once we release 5.0. In the mean time, Kibana 4.2 + x depends on ES 2.x.

When you do that, you should then upgrade the Kibana plugin for Marvel's UI to 2.3.1 (matching) and then it will read from the Marvel indices. Marvel 2.0 - 2.2 read from .marvel-es-data rather than .marvel-es-data-1 (and similar for the time series index).

Preferably:

  • Upgrade to ES 2.3.3
    • Install marvel-agent
    • Install license
  • Upgrade to Kibana 4.5.1
    • Install marvel [ui]

Hope that helps.

3

Thanks. Upgrading to 4.5.1 solved the issue.