How to install Marvel with free basic license and without Shield

Elastic Stack Elasticsearch
1

Hi folks,

I read through the information at https://www.elastic.co/guide/en/marvel/current/license-management.html, but I couldn't find a way to install my free basic license.

I have a 2 node elasticsearch cluster running ElasticSearch 2.0 on ubuntu 14.04. I tried installing the license plugin on one of the nodes, but this caused the ElasticSearch node to become offline - although the service was running, the node is not responding to requests and the log file had repeated messages:

failed to notify ClusterStateListener 
java.lang.IllegalStateException: master not available when registering auto-generated license
...
[my_index_1][[my_index_1][0]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: ShardNotFoundException[no such shard]; 
...
Caused by: [my_index_1][[my_index_1][0]] ShardNotFoundException[no such shard]
...

When I installed the license plugin on the other node as well, my whole cluster was down. The error on both nodes looked similar to what I enclosed above.

I tried install marvel-agent plugin on both nodes afterward. The errors persisted.

Am I missing anything?

I obtained the free basic license for Marvel, but I am not sure how I can install that with a cluster being down. Also, it seems that the instructions from the page applies for an installation with Shield. Can I install Marvel without Shield? If not, does Shield also have a free basic license?

I didn't know how to proceed, so I uninstalled both license and marvel-agent plugins from all my nodes and my cluster returns to green status.

Thanks,
Ian

2

Hi,

I encourage you to use the latest 2.1 version of elasticsearch. Once ES is installed (but not started), install the license and marvel-agent plugins (and configure marvel if needed).

Then start your nodes and check that your cluster is green or at least yellow. If it's red, checks the logs first.

Once the cluster is ok you can update your license. In your case it looks like the master wasn't available at the time you tried to PUT your license.

Marvel can work without Shield, the page you linked to mentions Shiel several times but it's only for users who have installed it in their cluster.

3

Hi Tanguy,

Thank you for the information. I will try the latest 2.1 version.

Ian

4

I am running into kind of the same problem.

I have a cluster with 3 nodes with automated master election. Preferably I do not bring the cluster down when installing marvel-agent. Is that even possible?

Whenever I install license and marvel-agent on one of the nodes, it cannot connect to master anymore. When I remove the plugins and restart the node, everything is OK again.

The node is giving this error:

[2016-02-18 13:00:50,037][INFO ][cluster.service ] [es02-ams01-tip] detected_master {es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300}, added {{es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300},{es01-ams01-tip}{7XJd-Dk6SfiINKM2t4xQKg}{10.10.30.1}{10.10.30.1:9300},}, reason: zen-disco-receive(from master [{es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300}]) [2016-02-18 13:00:50,055][INFO ][cluster.routing.allocation.decider] [es02-ams01-tip] updating [cluster.routing.allocation.enable] from [ALL] to [NONE] [2016-02-18 13:00:50,056][INFO ][indices.store ] [es02-ams01-tip] updating indices.store.throttle.max_bytes_per_sec from [10gb] to [150mb], note, type is [NONE] [2016-02-18 13:00:50,085][INFO ][discovery.zen ] [es02-ams01-tip] master_left [{es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300}], reason [transport disconnected] [2016-02-18 13:00:50,087][INFO ][discovery.zen ] [es02-ams01-tip] failed to send join request to master [{es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300}], reason [NodeDisconnectedException[[es03-ams01-tip][10.10.30.3:9300][internal:discovery/zen/join] disconnected]] [2016-02-18 13:00:50,422][WARN ][discovery.zen ] [es02-ams01-tip] master left (reason = transport disconnected), current nodes: {{es02-ams01-tip}{fLytB-HUS5eoYctNciIuSQ}{10.10.30.2}{10.10.30.2:9300},{es01-ams01-tip}{7XJd-Dk6SfiINKM2t4xQKg}{10.10.30.1}{10.10.30.1:9300},} [2016-02-18 13:00:50,423][INFO ][cluster.service ] [es02-ams01-tip] removed {{es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300},}, reason: zen-disco-master_failed ({es03-ams01-tip}{H0_Dfsv9RLOGNFsS2ARshQ}{10.10.30.3}{10.10.30.3:9300}) [2016-02-18 13:00:50,441][WARN ][cluster.service ] [es02-ams01-tip] failed to notify ClusterStateListener java.lang.IllegalStateException: master not available when registering auto-generated license

And the master is throwing this error:

[2016-02-18 13:00:50,619][DEBUG][transport.netty ] [es03-ams01-tip] connected to node [{es02-ams01-tip}{fLytB-HUS5eoYctNciIuSQ}{10.10.30.2}{10.10.30.2:9300}] [2016-02-18 13:00:50,711][DEBUG][cluster.service ] [es03-ams01-tip] processing [zen-disco-join(join from node[{es02-ams01-tip}{fLytB-HUS5eoYctNciIuSQ}{10.10.30.2}{10.10.30.2:9300}])]: execute [2016-02-18 13:00:50,711][DEBUG][cluster.service ] [es03-ams01-tip] cluster state updated, version [791], source [zen-disco-join(join from node[{es02-ams01-tip}{fLytB-HUS5eoYctNciIuSQ}{10.10.30.2}{10.10.30.2:9300}])] [2016-02-18 13:00:50,711][INFO ][cluster.service ] [es03-ams01-tip] added {{es02-ams01-tip}{fLytB-HUS5eoYctNciIuSQ}{10.10.30.2}{10.10.30.2:9300},}, reason: zen-disco-join(join from node[{es02-ams01-tip}{fLytB-HUS5eoYctNciIuSQ}{10.10.30.2}{10.10.30.2:9300}]) [2016-02-18 13:00:50,711][DEBUG][cluster.service ] [es03-ams01-tip] publishing cluster state version [791] [2016-02-18 13:00:50,780][WARN ][transport.netty ] [es03-ams01-tip] exception caught on transport layer [[id: 0x10065a67, /10.10.30.2:59175 => /10.10.30.3:9300]], closing connection java.lang.IllegalStateException: Message not fully read (request) for requestId [10], action [internal:plugin/license/cluster/register_trial_license], readerIndex [75] vs expected [75]; resetting

It seems that I have to install the plugins on all nodes and then bring the cluster down completely and then up again one-by-one. But, is there another way?

1 Like
5

Hi Bob,

Yes, you will need to do a full-cluster restart after installing Marvel.

Thanks,
Steve

1 Like
6

Thanks for sharing this info Bob!

My situation was very similar to yours, but not with Marvel or Shield --- it was with installing Watcher on our ES 2.3.1 cluster.

My mistake was that I installed the plugin on our single ES Client node and did not install it on the remaining Master and Data nodes. So when I restarted all ES nodes the cluster wouldn't come back online and kept failing (master not available messages in the log files).

To fix I stopped all ES services on all of the cluster nodes, installed the Watcher plugin on all nodes, then one by one manually started ES services by starting with Node1, then Node2, and so on. Only after doing this is when the cluster finally came back online.

So -- lessons learned here for me is this:

  1. If you make a change on one node member, you must do it on all of them.
  2. Read the manual, twice! Warnings about installing Watcher for a cluster says you must install on all member nodes. I missed this the first time around. - https://www.elastic.co/guide/en/watcher/current/installing-watcher.html

Joe

7

Thanks Bob - is that documented anywhere because if it is I completely missed it.