Limit to number of Nodes?

Elastic Stack Kibana
1

Is there a limit to the number of nodes displayed in the free version of Marvel?

Right now I have 12 data nodes, all showing up, but I have 12 logstash nodes as well with ES client nodes running on them. When I view Marvel I only see a few of the logstash nodes. The configs are identical because I copied them from one node to all others. I know the nodes are there because:
curl 10.1.55.2:/_cat_nodes | grep INDEXER
10.1.55.6 10.1.55.6 4 9 0.78 - - INDEXER_NODE_4
10.1.55.27 10.1.55.27 1 9 0.88 - - INDEXER_NODE_9
10.1.55.30 10.1.55.30 0 9 0.59 - - INDEXER_NODE_12
10.1.55.25 10.1.55.25 1 9 0.56 - - INDEXER_NODE_7
10.1.55.26 10.1.55.26 2 9 0.35 - - INDEXER_NODE_8
10.1.55.24 10.1.55.24 3 9 0.81 - - INDEXER_NODE_6
10.1.55.28 10.1.55.28 2 9 0.48 - - INDEXER_NODE_10
10.1.55.4 10.1.55.4 0 9 0.68 - - INDEXER_NODE_2
10.1.55.23 10.1.55.23 3 9 0.29 - - INDEXER_NODE_5
10.1.55.29 10.1.55.29 1 9 0.31 - - INDEXER_NODE_11
10.1.55.5 10.1.55.5 3 9 0.66 - - INDEXER_NODE_3

This shows all the indexer nodes I have running right now. (I know node 1 is not listed it is currently offline.)

Marvel shows

Capture.PNG3126×1114 144 KB

Why would all of the nodes be listed when running the curl command but not show up in Marvel.

I have confirmed I am running the same version of elasticsearch, logstash, and marvel-agent on all nodes
ES 2.3.1
LS 2.3.1
Marvel-Agent 2.3.1
License 2.3.1

Thanks
Tim

2

Hi Tim,

I have not seen this issue and there's no realistic limit that is set.

On the cluster being monitored, can you run:

$ curl -XGET host:9200/_cat/plugins?v

This will show what nodes believe to be installed as opposed to what's on the file system. It should list 12 data nodes, 12 client nodes (your "indexers"), and presumably 3 master nodes. Each one should show marvel-agent installed for the same version.

Let me know.

3 
name                     component    version type url
KIBANA_SEARCH_BALANCER_1 license      2.3.3   j
KIBANA_SEARCH_BALANCER_1 marvel-agent 2.3.3   j
MASTER_NODE_1            license      2.3.1   j
MASTER_NODE_1            marvel-agent 2.3.1   j
INDEXER_NODE_5           license      2.3.1   j
INDEXER_NODE_5           marvel-agent 2.3.1   j
INDEXER_NODE_8           license      2.3.1   j
INDEXER_NODE_8           marvel-agent 2.3.1   j
WORKER_NODE_12           license      2.3.1   j
WORKER_NODE_12           marvel-agent 2.3.1   j
WORKER_NODE_7            license      2.3.1   j
WORKER_NODE_7            marvel-agent 2.3.1   j
WORKER_NODE_10           license      2.3.1   j
WORKER_NODE_10           marvel-agent 2.3.1   j
KIBANA_SEARCH_BALANCER_2 license      2.3.3   j
KIBANA_SEARCH_BALANCER_2 marvel-agent 2.3.3   j
WORKER_NODE_5            license      2.3.1   j
WORKER_NODE_5            marvel-agent 2.3.1   j
WORKER_NODE_6            license      2.3.1   j
WORKER_NODE_6            marvel-agent 2.3.1   j
PB2ES_NODE_1             license      2.3.3   j
PB2ES_NODE_1             marvel-agent 2.3.3   j
INDEXER_NODE_6           license      2.3.1   j
INDEXER_NODE_6           marvel-agent 2.3.1   j
INDEXER_NODE_3           license      2.3.1   j
INDEXER_NODE_3           marvel-agent 2.3.1   j
WORKER_NODE_9            license      2.3.1   j
WORKER_NODE_9            marvel-agent 2.3.1   j
WORKER_NODE_11           license      2.3.1   j
WORKER_NODE_11           marvel-agent 2.3.1   j
WORKER_NODE_1            license      2.3.1   j
WORKER_NODE_1            marvel-agent 2.3.1   j
WORKER_NODE_8            license      2.3.1   j
WORKER_NODE_8            marvel-agent 2.3.1   j
MASTER_NODE_3            license      2.3.1   j
MASTER_NODE_3            marvel-agent 2.3.1   j
INDEXER_NODE_10          license      2.3.1   j
INDEXER_NODE_10          marvel-agent 2.3.1   j
INDEXER_NODE_12          license      2.3.1   j
INDEXER_NODE_12          marvel-agent 2.3.1   j
INDEXER_NODE_11          license      2.3.1   j
INDEXER_NODE_11          marvel-agent 2.3.1   j
WORKER_NODE_2            license      2.3.1   j
WORKER_NODE_2            marvel-agent 2.3.1   j
MASTER_NODE_2            license      2.3.1   j
MASTER_NODE_2            marvel-agent 2.3.1   j
WORKER_NODE_4            license      2.3.1   j
WORKER_NODE_4            marvel-agent 2.3.1   j
INDEXER_NODE_4           license      2.3.1   j
INDEXER_NODE_4           marvel-agent 2.3.1   j
INDEXER_NODE_9           license      2.3.1   j
INDEXER_NODE_9           marvel-agent 2.3.1   j
INDEXER_NODE_2           license      2.3.1   j
INDEXER_NODE_2           marvel-agent 2.3.1   j
WORKER_NODE_3            license      2.3.1   j
WORKER_NODE_3            marvel-agent 2.3.1   j
INDEXER_NODE_7           license      2.3.1   j
INDEXER_NODE_7           marvel-agent 2.3.1   j
PB2ES_NODE_2             license      2.3.3   j
PB2ES_NODE_2             marvel-agent 2.3.3   j
4

It certainly looks like it's installed everywhere, but it's noteworthy that some nodes are running slightly different versions.

Now that we have that plugin output, can we correlate that with the nodes info stored in your index?

$ curl -XGET "host:9200/.marvel-es-data-1?pretty&size=50" -o data-file.json

Then just attach the output either here or as a gist (posting it as a gist makes it easier for you to delete later).

At the same time, can you also add any configuration settings that you are applying to one of the nodes that is missing and one of the same types that is not missing?

Thanks

5

So I'm not sure what happened but after 2 days of not seeing some of the nodes in marvel now I am seeing all of them. Its strange. I made no configuration changes at all.

I'm going to chalk it up to having to wait for some reason.

Thanks for the help. I am still happy to provide the requested info if it will help you troubleshoot this issue. But I am considering it resolved.

Thanks again

6

I appreciate the offer, but since it's all appearing I think it's too late to see what was failing.

I'm glad that it kicked in!

Thanks,
Chris