License not working on 2.1 offline server

Elastic Stack Elasticsearch
1

I just upgraded elasticsearch to 2.1.1 and kibana to 4.3. on a server which is offline

I'm trying to install watcher and license but am having trouble - it looks largely to do with license.

When installing all goes fine, apart from "unable to verify checksum for downloaded plugin" - which considering we are offline I reckon is fine.

But then testing watcher gives me an error (curl -XGET 'http://localhost:9200/_watcher'').

Looking into the logs a bit further I have a lot of errors as below:

java.lang.IllegalStateException: Message not fully read (request) for requestId [5425], action [internal:plugin/license/cluster/register_trial_license], readerIndex [75] vs expected [75]; resetting
<snip>
[2016-01-13 17:00:50,261][INFO ][discovery.zen            ] [elasticsearch] master_left [{elasticsearch}{KyAtACrETo-sDYxR8IQL8w}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
[2016-01-13 17:00:50,261][WARN ][discovery.zen            ] [elasticsearch] master left (reason = transport disconnected), current nodes: {{elasticsearch}{phMcRNyoSTuFqAj4xSAUqg}{127.0.0.1}{127.0.0.1:9301},}
[2016-01-13 17:00:50,262][INFO ][cluster.service          ] [elasticsearch] removed {{elasticsearch}{KyAtACrETo-sDYxR8IQL8w}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-master_failed ({elasticsearch}{KyAtACrETo-sDYxR8IQL8w}{127.0.0.1}{127.0.0.1:9300})
[2016-01-13 17:00:50,284][WARN ][cluster.service          ] [elasticsearch] failed to notify ClusterStateListener
java.lang.IllegalStateException: master not available when registering auto-generated license
<snip>

Note I've snipped a lot off the exceptions above to fit in the character limit. But I think the headlines give you the general gist. If anyone needs more just ask.

To me this means license isn't working properly for me.

Any ideas?

Sorry if this is in the wrong topic, but it feels more general than a specific plugin.

2

Any one have any ideas?

I've been looking into it most of today and haven't found anything.

Best idea I have is that license is trying to connect to something to generate a license - which considering this is an offline install would make offline installations pretty broken. So I have my doubts on it.

3

Not fully sure what is going on but not having an elected master means you don't have a cluster..

4

Odd. I'm definitely seeing data come through on Kibana.

How can I check/assign a master?

5

Note I only have a single node.

6

Checking the cluster state, gives the master_node having the same uuid as in the error message.

Cluster health status is put down as yellow.

7

This is quite odd.. can you verify that there aren't multiple nodes running on your machine, perhaps accidentally? Did you restart Elasticsearch after installing the plugins?

Also, I think we'll need to see the full log to understand what might have happened here. Can you use pastebin or a gist to share the longer log?

8

At home so can't confirm till I get back into the office. I will pastebin the log tomorrow.

Definitely restarted Elasticsearch after installing. AFAIK there are not multiple nodes running, it's all off a single isolated VM right now as we're just experimenting.

In all fairness I was not the one to setup the VM so I'm not 100% on all the setup. I'm making a fresh install at home tonight to get a better feel for it.

9

I, probably quite stupidly, decided to re-install es, but as a service. Which didn't work. reverted to the tar.gz distribution and now have different problems mentioned at: Elasticsearch red indices on upgrade

I have a mild theory that as all the indices were yellow the cluster couldn't form which in turn may have caused the watcher problem.

I reckon this may be because replicas were set to some number higher than was possible (considering the single node). I had more yellow indices after re-installing and setting number of replicas to 0 allowed them to turn green. But can't prove it would have solved the issue.

10

Working now. I reckon I had to get all the indices green. which I have done (by deleting red ones...).

11

I'm glad you are up and running!