Waiting for Marvel Data.... No Data

Elastic Stack Kibana
1

While trying to resolve an issue where the node names where showing up as the node ID numbers it was suggested that I remove the Marvel indexes and the .marvel-es-data file. After doing so data is no longer showing up in Marvel. I receive the message:

Waiting for Marvel Data
It appears that we have not received any data for this cluster. Marvel data is sent every 10 seconds by the agent plugin; once the data appears this page will automatically redirect to the overview page

.marvel-es-data is no longer listed as an index, but there is a .marvel-es-data-1

curl -XGET http://p-es-1:9200/_cat/indices?v

green  open   .marvel-es-data-1         1   1          9          290       55kb         27.3kb
green  open   .marvel-es-1-2016.05.20   1   1      70242          450     66.5mb         33.2mb
green  open   .marvel-es-1-2016.05.21   1   1     280214         1264    287.9mb        143.8mb

So im looking for a little insight on how to get Marvel back and running as normal.

1 Like
2

Anything in your ES logs?

3

I am not able to find anything within the ES logs related any issues with Marvel

4

What version of Kibana and the Marvel UI plugin are you running? You are running ES 2.3.x based on those Marvel index names.

It should be Kibana 4.5.x and Marvel UI 2.3.x based on those index names.

5

ES and Marvel are all 2.3 and Kibana is 4.3.1

Marvel didnt have any issues with Kibana 4.3.1 until I tried to resolve the node name issue by removing the Marvel indexes and the .marvel-es-data file

6

I am still looking for a little guidance with this issue. Marvel has not been viewable for over a week now. I can see that indexes are still being created, so it seems to just be related to something with the UI.

7

Jason,

Sorry! I didn't see your initial response.

The issue is that you're using the wrong version of Kibana. Elasticsearch 2.3.x should be used with Kibana 4.5.y (generically, the version of Kibana is the version of ES + 2.2; so ES 2.0 requires Kibana 4.2, ES 2.1 requires Kibana 4.3, etc.). The plugin for the Marvel UI, should match exactly what you're using with Elasticsearch, which itself should match exactly the version of Elasticsearch.

As you're running ES 2.3.x, you need to run the marvel-agent plugin version 2.3.x (you will be because of how ES plugins work).

From there, you should upgrade Kibana to 4.5.1 (latest) and install the associated version of the marvel UI plugin for it associated with ES 2.3.x. Assuming you're running ES 2.3.1, then you could install it as:

$ bin/kibana plugin --install elasticsearch/marvel/2.3.1

Settings should stay the same. This also provides an updated version of Kibana with a security fix for node.js, which is bundled with it.

You can quickly check your version of ES by checking:

$ curl -XGET p-es-1:9200/

This should provide the version.number to indicate what you need to grab for Kibana.

8

As a note: this versioning craziness will go away with Kibana 5 and Elasticsearch 5. Once we release those versions, all versions will be in alignment, from the plugins to the applications themselves.

9

Thanks for getting back to me Chris. So I did as you suggested and removed and reinstalled the version to match Elasticsearch. Here is the results of the install.

-> Installing marvel-agent...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.3.3/marvel-agent-2.3.3.zip ...
Downloading ..........DONE
Verifying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel-agent/2.3.3/marvel-agent-2.3.3.zip checksums if available ...
Downloading .DONE
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission setFactory
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
Installed marvel-agent into /usr/share/elasticsearch/plugins/marvel-agent

As you can see its isntalling version 2.3.3, which matches the version of ES that I am running.

{
  "name" : "p-es-1",
  "cluster_name" : "Elasticsearch-Cluster-1",
  "version" : {
    "number" : "2.3.3",
    "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
    "build_timestamp" : "2016-05-17T15:40:04Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

I am still getting the message that Marvel is waiting for data.

10

You restarted ES, right?

11

Hi Jason,

I think the confusion here might be around Marvel itself, and particularly how it changed in version 1.x to 2.x.

  • In Marvel 1.x, it was both the UI and the "agent" that collected data within Elasticsearch itself.
  • In Marvel 2.x, it is split into two plugins:
    • marvel-agent as you just installed is for collecting Elasticsearch data and sending it somewhere.
    • marvel UI, which is the Kibana plugin.

The issue here appears to be entirely related to the UI, which relates solely to the marvel UI plugin and Kibana itself. The marvel-agent sounds like it is/was working (which as Mark noted, requires a restart to take effect, but I expect it to not change anything).

Can you try setting up an instance of Kibana 4.5.1 with the marvel UI plugin? You don't have to replace your existing installation until you confirm that it fixes your issue.

Let me know.

12

I tried removing Marvel and not Marvel-Plugin with the following result

-> Removing marvel...
Plugin marvel not found. Run "plugin list" to get list of installed plugins.

I tried to see if I needed to reinstall it with the following result

-> Installing marvel...
Trying https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel/2.3.3/marvel-2.3.3.zip ...
Failed: FileNotFoundException[https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel/2.3.3/marvel-2.3.3.zip]; nested: FileNotFoundException[https://download.elastic.co/elasticsearch/release/org/elasticsearch/plugin/marvel/2.3.3/marvel-2.3.3.zip];
ERROR: failed to download out of all possible locations..., use --verbose to get detailed information

Rebooting didnt seem to resolve the issues either.

13

Jason,

The marvel UI plugin is in Kibana, not Elasticsearch.

You need to mess with Kibana by installing Kibana 4.5.1 and the marvel plugin there following the instructions above.

Hope that helps.

14

Oh I see, im running a incapable version of Kibana. I guess I never added the Kibana repo so it isnt getting any updates. I originally installed Kibana manually and just now added the repo following these instructions:

https://www.elastic.co/guide/en/kibana/4.5/setup.html#setup-repositories

Yum is still not seeing that there is an update for Kibana. Do I need to install the update manually?

15

Hi Jason,

Sorry for the delay.

But yes, you should just override the install by removing the existing installation, then installing the updated version on top of it. The Kibana repo is a little unusual in that it changes the repository with each release (note that it adds 4.5 to the repository name). This technically means that they're not really associated as well as they could be (e.g., Elasticsearch's RPM / DEB packages use 2.x instead of having one for 2.0, 2.1, 2.2, etc.).

This will be fixed in 5.x where there repo will be "5.x" instead of "5.0", "5.1", etc.

Hope that helps.

16

Won't I lose all existing dashboards and widgets if I uninstall the current version?

Jason Petty
Systems Administrator
Information Technology
UC Davis School of Veterinary Medicine
530-752-2271 (o)

17

No. Kibana does not store things in its local file store. Instead, it uses Elasticsearch as its data store, so upgrading is as simple as backing up any configuration changes that you've made and re-installing any Kibana plugins (most plugins are not expected to work across minor releases, such as 4.3 to 4.4 or 4.5).

Conveniently, you can test this out by installing Kibana 4.5 manually (download the tar and extract it) without touching Kibana 4.3. As long as Kibana 4.3 is not running, then you can quickly confirm that you still have all of the dashboards by connecting to the same Elasticsearch cluster (you can confirm it even if Kibana 4.3 is running, but then you need to either install it on another machine, which is fine(!) or change its startup port from the default of 5601).

19

I also had the problem on my cluster and came up with a quick fix. Basically, The latest version of Elasticsearch plugin marvel-agent creates index named ".marvel-es-data-1", and old versions of Kibana marvel plugin read from ".marvel-es-data". So it cannot find any marvel data and the dashboard won't show.

Adding an alias ".marvel-es-data" and point it to ".marvel-es-data-1" will recover your dashboard:

curl -XPOST "http://p-es-1:9200/_aliases" -d '
{
  "actions": {
    {
      "add": {
        "index": ".marvel-es-data-1",
        "alias": ".marvel-es-data"
      }
    }
  }
}
'
20

Thanks Yang for posting the work-around solution!

21

Hi Yang,

Sorry if this is a newbie question, im still very new to ELK. So when I run that command I get the following error:

{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}