Marvel reflect changes very slowly

Elastic Stack Elasticsearch
1

I just upgraded elasticsearch from 2.1 to 2.3, in my elasticsearch cluster, there is 1 master&data node, 1 data node, and one client node. Marvel is installed in all the three nodes.
I found that Marvel reflect index changes very slowly, whether inserting documents or creating/deleting index, I cannot see the changes from Marvel few hours later.
For example, I used the command line to delete the index named elkstats_other_sources

curl -XDELETE Melasticsearch:9200/elkstats_other_sources
curl -XGET Melasticsearch:9200/elkstats_other_sources/_stats/docs?pretty=true

The response on command line is correct:

{
"error" : {
"root_cause" : [ {
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "elkstats_other_sources",
"index" : "elkstats_other_sources"
} ],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "elkstats_other_sources",
"index" : "elkstats_other_sources"
},
"status" : 404
}

However, on Marvel, I can still see the index active for a few hours.

1643×455 29.3 KB

Another example about Marvel is slow to reflect the status during index inserting
I started importing data from 10:30am

826×265 10.7 KB

Marvel shows the data inserting started at 11:35am

817×418 21.4 KB

2

Kibana, allow to see historic on the top right corner of your screen. Change the interval to 5 minutes ... Normally it would show only the remaining indices in your cluster.

Hope it helps !

Nicolas.

3

Thanks for reply, but I don't think it is related to the left corner choice. I allow kibana and Marvel to show the latest 1 hour, it should reflect the current changes of the index.

4

If you set to 1 hour and your change has been done 56 minute ago, normally you should still see it.

5

I did the experiment again, I deleted testing index using
curl -XDELETE Melasticsearch:9200/testing

And I set in marvel to show the latest 30 second

Pasted image1665×203 11.2 KB

I am pretty sure that I already deleted more than 5 minutes ago, but this index is still showing as active.

Pasted image1641×429 27.8 KB

I was using elasticsearch 2.1X, actually when I delete an index, almost immediately, It shows N/A status on marvel.

6

Actually, I apologize for not convey it correctly, Marvel is slow, Kibana is not slow.

7

If both Marvel and Kibana are seeing it, then it still exists somewhere. Kibana in particular fetches things dynamically; it's not maintaining that index somewhere.

On the other hand, Marvel definitely does maintain information. However, it should show "N/A" within the Marvel Agent's interval (defaults to 10s + the webpage's request interval) to note that it's not getting anything else. You'd also notice that the color to the left of the index name disappears (the green block in the highlighted portion).

I was using elasticsearch 2.1X, actually when I delete an index, almost immediately, It shows N/A status on marvel.

Marvel 2.1 and Marvel 2.3 use a different index for reading/writing cluster details. Your instance of Kibana that you are using to monitor the cluster needs to be running the same instance of the plugin as a result.

Marvel 2.1 requires:

  • ES 2.1.x
  • Kibana 4.3
  • Marvel UI 2.1.x

Marvel 2.3.x requires:

  • ES 2.3.x
  • Kibana 4.5
  • Marvel UI 2.3.x

Perhaps this is your issue?

8

How do I check Marvel versions? I actually installed Marvel after installing Elasticsearch2.3 just using the command line:
sudo bin/plugin install marvel-agent

So I think it should automatically pull the proper version of Marvel.

Do I need to manually download and install Marvel 2.3?

9

Hi Sharon,

So there are two components to Marvel starting with 2.x:

  • marvel-agent
  • marvel UI

The first one must match the version of ES exactly, and the way that you installed it guarantees that to be true. You can confirm this by checking

$ curl -XGET host:9200/_cat/plugins?v

The Marvel UI plugin is a plugin for Kibana and its plugin version's are not quite as strict yet, so it's less obvious what you might have installed (or not installed) on that side.

To check the Marvel UI plugin version, you have to check a file in the Kibana directory:

$ grep -n version ${KIBANA_DIR}/installedPlugins/marvel/package.json

If those two versions don't match, then reinstall the Marvel UI plugin so that it does match.

10

My marvel kibana plugin version is 2.2.0 I will upgrade it, really appreciate your help!