Errors with Marvel indexes when starting second node

Elastic Stack Elasticsearch
1

Hello,

I have a master node and a data node. When I add indexes to my master node and start my data node, I get the errors (BroadcastShardOperationFailedException, IllegalIndexShardStateException). It still sync the indexes and everything seems to be working properly but I would like to fix the errors if possible.

I am new to Elasticsearch so any guidance would be gladly appreciated.

Here are the logs of what's happening exactly :

Error when starting the second node : http://pastebin.com/trD5hiNv
Error when disconnecting the second node : http://pastebin.com/Nb8vvEbg

Note : I have been doing tests on my master node with the data node offline adding multiple indexes using a large file in bulk (50mbs file). The problem seems to happen only when I have indexes other than the ones marvel create. If I delete all indexes but the ones from marvel, I don't get any errors. If I add indexes and restart nodes, the errors come back.

Thank you!

2

Hi,

The exceptions occurs because the marvel agent requires the data node to be up. Marvel queries the data node to get stats about indices and shards and if you did not configure HTTP exporters marvel will also try to index data locally.

There's a pull request (https://github.com/elastic/elasticsearch/pull/14950) on elasticsearch to log at TRACE (and not DEBUG) level some of the exceptions you have.

In case of local exporter, marvel requires data node because it index data. One thing you can do it start with a higher sampling interval (configuration: ̀marvel.agent.interval: 30s`) and once your cluster & nodes are up you update it using the Cluster Update API.

-- Tanguy