Our company has decided to use ElasticSearch for saving some critical data
. We are using Marvel for real-time monitoring the cluster.
Specifically we have a 9 node cluster with 3 Master Nodes, 2 Search Load
Balancers and 4 Data Nodes . Additionally we have a 2 node cluster which
has Marvel Indexes and is being used for monitoring the main cluster.
We are facing various issues which are as follows :
a)Although both the clusters (Production as well as Monitoring) are in the
same Subnet still any changes made to the production cluster is not
reflected immediately in Marvel Dashboard. Hence we are not able to do a
real-time monitoring of the cluster.
b)Although we are not deleting a single document still the DELETED DOCUMENTS graph
on the node statistics dashboard shows spikes which is again a concern for
us.
For the reference I am attaching the screenshots please find the same in
the attachments.
Please can someone help with the above mentioned issues.
Our company has decided to use Elasticsearch for saving some critical data
. We are using Marvel for real-time monitoring the cluster.
Specifically we have a 9 node cluster with 3 Master Nodes, 2 Search Load
Balancers and 4 Data Nodes . Additionally we have a 2 node cluster which
has Marvel Indexes and is being used for monitoring the main cluster.
We are facing various issues which are as follows :
a)Although both the clusters (Production as well as Monitoring) are in the
same Subnet still any changes made to the production cluster is not
reflected immediately in Marvel Dashboard. Hence we are not able to do a
real-time monitoring of the cluster.
b)Although we are not deleting a single document still the DELETED
DOCUMENTS graph on the node statistics dashboard shows spikes which is
again a concern for us.
For the reference I am attaching the screenshots please find the same in
the attachments.
Please can someone help with the above mentioned issues.
Our company has decided to use Elasticsearch for saving some critical
data . We are using Marvel for real-time monitoring the cluster.
Specifically we have a 9 node cluster with 3 Master Nodes, 2 Search Load
Balancers and 4 Data Nodes . Additionally we have a 2 node cluster which
has Marvel Indexes and is being used for monitoring the main cluster.
We are facing various issues which are as follows :
a)Although both the clusters (Production as well as Monitoring) are in
the same Subnet still any changes made to the production cluster is not
reflected immediately in Marvel Dashboard. Hence we are not able to do a
real-time monitoring of the cluster.
b)Although we are not deleting a single document still the DELETED
DOCUMENTS graph on the node statistics dashboard shows spikes which is
again a concern for us.
For the reference I am attaching the screenshots please find the same in
the attachments.
Please can someone help with the above mentioned issues.
Hi Mark,
Thanks for the reply.I have mentioned in my post that we have DECIDED to use ,actually we have still not started using it in production .What I mentioned is just a prototype that we have decided to use in production and doing the final non-functional testing on the developer side.I read it on the site that we have to purchase license when we start using it commercially right ?
And the problem is that while testing if we shut-down any of the data nodes or master's or SLB's ,for around an hour or so, Marvel still shows that those nodes are available as it is visible in the screenshot. As we need a near real-time monitoring so this will not help.
Hi Ivan,
Thanks for the reply.We are not deleting any of the documents what we are doing is just inserting records and generating reports by using aggregation API.As I mentioned that the data we are putting in ElasticSearch is critical and we can't afford to loose it ,that's why Marvel showing spikes in the DELETED DOCUMENTS graph is of concern.
Are you updating documents? when reindex a document lucene deletes the old
copies and indexes new ones. Those deletes count for the chart. After a
merge happens they are removed.
Regarding shutting down nodes for >1h. When marvel doesn't hear from a node
for more than 1m, the node's data is considered stale and the row will be
grayed out with an information icon over which you can hover and see how
long the node has not reported. I see that your screenshot. What's
interesting is that after 10m those should be removed completed. Was that
not the case? what version of Marvel are you using?
Cheers,
Boaz
On Friday, April 4, 2014 6:30:45 AM UTC+2, AryanJ wrote:
Hi Mark,
Thanks for the reply.I have mentioned in my post that we have DECIDED to
use
,actually we have still not started using it in production .What I
mentioned
is just a prototype that we have decided to use in production and doing
the
final non-functional testing on the developer side.I read it on the site
that we have to purchase license when we start using it commercially right
?
And the problem is that while testing if we shut-down any of the data
nodes
or master's or SLB's ,for around an hour or so, Marvel still shows that
those nodes are available as it is visible in the screenshot. As we need a
near real-time monitoring so this will not help.
Hi Boaz,
Thanks for the reply . We are not deleting or updating any documents . Actually we are able to figure that out .What is happening is that Deleted Documents Graph shows some spike for a particular node whenever we stop that node. We are assuming that when a node goes down the shards are moved around for re-balancing, hence data would be deleted from one node and moved to other this is what shows spikes.
It takes some time to update the status of the the nodes.I am not sure if it is exactly 10 min or not.
I am not sure about the marvel version what is know is that it should be the latest one ..as we installed it using bin/plugin -i elasticsearch/marvel/latest
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
what is know is that it should be the latest one ..as we installed it using bin/plugin -i elasticsearch/marvel/latest