Issues with Marvel and Shield

Elastic Stack Elasticsearch
1

I recently restarted some nodes in our cluster after a configuration change and unfortunately they aren't coming back up cleanly. There appears to be an issue with marvel. There is a reference to __marvel_user however I never configured a user by this name. It may have been that we installed marvel before shield so this wasn't required? I don't have anything configured for exporters as this is not a monitoring cluster. Should I?

Caused by: ElasticsearchException[failure in bulk execution:
[0]: index [.marvel-es-1-2016.09.01], type [node_stats], id [AVbmWt6A4696eKrladi 9], message [RemoteTransportException[[iot-prod-5][10.19.17.84:9300][indices:dat a/write/bulk[s]]]; nested: ElasticsearchSecurityException[action [indices:data/w rite/bulk[s]] is unauthorized for user [__marvel_user]];]]
at org.elasticsearch.marvel.agent.exporter.local.LocalBulk.flush (LocalBulk.java:118)

2

Hi Nate,

Can you provide the output of

$ curl -XGET localhost:9200/_cat/plugins?v

Thanks,
Chris

3 
name       component       version type url
iot-prod-1 license         2.3.4   j
iot-prod-1 repository-hdfs 2.3.3   j
iot-prod-1 shield          2.3.4   j
iot-prod-1 watcher         2.3.4   j
iot-prod-6 license         2.3.3   j
iot-prod-6 repository-hdfs 2.3.2   j
iot-prod-6 shield          2.3.3   j
iot-prod-5 license         2.3.3   j
iot-prod-5 repository-hdfs 2.3.2   j
iot-prod-5 shield          2.3.3   j
iot-prod-3 license         2.3.3   j
iot-prod-3 repository-hdfs 2.3.2   j
iot-prod-3 shield          2.3.3   j
iot-prod-4 license         2.3.3   j
iot-prod-4 repository-hdfs 2.3.2   j
iot-prod-4 shield          2.3.3   j

I currently have marvel uninstalled

4

Hi Nate,

So I don't see the marvel-agent anywhere in there. Is this happening from another cluster?

I currently have marvel uninstalled

That makes more sense.

Thanks,
Chris

5

Yea, it was keeping the cluster from coming up so I just removed it for now but I obviously need to get it working again.

6

Which version of Elasticsearch is running on those nodes?

They all have different versions of plugins, which implies different versions of Elasticsearch. You should always have the same version of the plugin as the running version of ES.

I am assuming that your environment has two versions running: 2.3.4 on one node and 2.3.3 on another.

Let me know

7

It looks like we have a client node which is running 2.3.4. The rest are running 2.3.3

8

I verified that the version of the plugin is 2.3.3 which is the same as the version of ES running on that node. At the moment this errors is just repeating in the log. It hints that __marvel_user doesn't have permissions to create a new index. I never had to configure this user previously.

Any ideas as to what this user is?