Marvel Shows Documents deleted


(BornGenius) #1

Hi all,

Our company has decided to use ElasticSearch for saving some critical data
. We are using Marvel for real-time monitoring the cluster.

Specifically we have a 9 node cluster with 3 Master Nodes, 2 Search Load
Balancers and 4 Data Nodes . Additionally we have a 2 node cluster which
has Marvel Indexes and is being used for monitoring the main cluster.
We are facing various issues which are as follows :

a)Although both the clusters (Production as well as Monitoring) are in the
same Subnet still any changes made to the production cluster is not
reflected immediately in Marvel Dashboard. Hence we are not able to do a
real-time monitoring of the cluster.

b)Although we are not deleting a single document still the DELETED DOCUMENTS graph
on the node statistics dashboard shows spikes which is again a concern for
us.

For the reference I am attaching the screenshots please find the same in
the attachments.

Please can someone help with the above mentioned issues.

Thanks

AryanJ

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/465b5378-0ee2-4720-96d8-f111cba5d7bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

FYI you are required to license Marvel with that number of nodes -
http://www.elasticsearch.com/marvel

What sort of time differences are you seeing between changes and the
display?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 3 April 2014 23:43, AryanJ aman_aryan_jolly@yahoo.com wrote:

Hi all,

Our company has decided to use ElasticSearch for saving some critical data
. We are using Marvel for real-time monitoring the cluster.

Specifically we have a 9 node cluster with 3 Master Nodes, 2 Search Load
Balancers and 4 Data Nodes . Additionally we have a 2 node cluster which
has Marvel Indexes and is being used for monitoring the main cluster.
We are facing various issues which are as follows :

a)Although both the clusters (Production as well as Monitoring) are in the
same Subnet still any changes made to the production cluster is not
reflected immediately in Marvel Dashboard. Hence we are not able to do a
real-time monitoring of the cluster.

b)Although we are not deleting a single document still the DELETED
DOCUMENTS graph on the node statistics dashboard shows spikes which is
again a concern for us.

For the reference I am attaching the screenshots please find the same in
the attachments.

Please can someone help with the above mentioned issues.

Thanks

AryanJ

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/465b5378-0ee2-4720-96d8-f111cba5d7bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/465b5378-0ee2-4720-96d8-f111cba5d7bb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YTo7CwHQV5Ct4n1Qx1Y440GqM7Vr0--7GCBgwovhBxKA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #3

Are you updating documents? Lucene does not update documents directly, but
does a delete/insert.

--
Ivan

On Thu, Apr 3, 2014 at 2:27 PM, Mark Walkom markw@campaignmonitor.comwrote:

FYI you are required to license Marvel with that number of nodes -
http://www.elasticsearch.com/marvel

What sort of time differences are you seeing between changes and the
display?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 3 April 2014 23:43, AryanJ aman_aryan_jolly@yahoo.com wrote:

Hi all,

Our company has decided to use ElasticSearch for saving some critical
data . We are using Marvel for real-time monitoring the cluster.

Specifically we have a 9 node cluster with 3 Master Nodes, 2 Search Load
Balancers and 4 Data Nodes . Additionally we have a 2 node cluster which
has Marvel Indexes and is being used for monitoring the main cluster.
We are facing various issues which are as follows :

a)Although both the clusters (Production as well as Monitoring) are in
the same Subnet still any changes made to the production cluster is not
reflected immediately in Marvel Dashboard. Hence we are not able to do a
real-time monitoring of the cluster.

b)Although we are not deleting a single document still the DELETED
DOCUMENTS graph on the node statistics dashboard shows spikes which is
again a concern for us.

For the reference I am attaching the screenshots please find the same in
the attachments.

Please can someone help with the above mentioned issues.

Thanks

AryanJ

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/465b5378-0ee2-4720-96d8-f111cba5d7bb%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/465b5378-0ee2-4720-96d8-f111cba5d7bb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624YTo7CwHQV5Ct4n1Qx1Y440GqM7Vr0--7GCBgwovhBxKA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624YTo7CwHQV5Ct4n1Qx1Y440GqM7Vr0--7GCBgwovhBxKA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDP60zh3Bhst3k2YdWA1ayS%3DR7k6bVgNdG1_ss8Mp6r_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(BornGenius) #4

Hi Mark,
Thanks for the reply.I have mentioned in my post that we have DECIDED to use ,actually we have still not started using it in production .What I mentioned is just a prototype that we have decided to use in production and doing the final non-functional testing on the developer side.I read it on the site that we have to purchase license when we start using it commercially right ?

And the problem is that while testing if we shut-down any of the data nodes or master's or SLB's ,for around an hour or so, Marvel still shows that those nodes are available as it is visible in the screenshot. As we need a near real-time monitoring so this will not help.

Thanks
AryanJ


(BornGenius) #5

Hi Ivan,
Thanks for the reply.We are not deleting any of the documents what we are doing is just inserting records and generating reports by using aggregation API.As I mentioned that the data we are putting in ElasticSearch is critical and we can't afford to loose it ,that's why Marvel showing spikes in the DELETED DOCUMENTS graph is of concern.

Thanks
AryanJ


(Boaz Leskes) #6

Hi Arjan,

Are you updating documents? when reindex a document lucene deletes the old
copies and indexes new ones. Those deletes count for the chart. After a
merge happens they are removed.

Regarding shutting down nodes for >1h. When marvel doesn't hear from a node
for more than 1m, the node's data is considered stale and the row will be
grayed out with an information icon over which you can hover and see how
long the node has not reported. I see that your screenshot. What's
interesting is that after 10m those should be removed completed. Was that
not the case? what version of Marvel are you using?

Cheers,
Boaz

On Friday, April 4, 2014 6:30:45 AM UTC+2, AryanJ wrote:

Hi Mark,
Thanks for the reply.I have mentioned in my post that we have DECIDED to
use
,actually we have still not started using it in production .What I
mentioned
is just a prototype that we have decided to use in production and doing
the
final non-functional testing on the developer side.I read it on the site
that we have to purchase license when we start using it commercially right
?

And the problem is that while testing if we shut-down any of the data
nodes
or master's or SLB's ,for around an hour or so, Marvel still shows that
those nodes are available as it is visible in the screenshot. As we need a
near real-time monitoring so this will not help.

Thanks
AryanJ

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Marvel-Shows-Documents-deleted-tp4053389p4053483.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/182f1197-74c2-4c7e-a52e-18fefb6e2345%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(BornGenius) #7

Hi Boaz,
Thanks for the reply . We are not deleting or updating any documents . Actually we are able to figure that out .What is happening is that Deleted Documents Graph shows some spike for a particular node whenever we stop that node. We are assuming that when a node goes down the shards are moved around for re-balancing, hence data would be deleted from one node and moved to other this is what shows spikes.

It takes some time to update the status of the the nodes.I am not sure if it is exactly 10 min or not.
I am not sure about the marvel version :frowning: what is know is that it should be the latest one ..as we installed it using bin/plugin -i elasticsearch/marvel/latest

Where can i find that ?????


(system) #8