Unsure of growth of index when nothing happening (clean install)


(Grant Christensen) #1

Hi all, first time Elasticsearch user.

I am using Elastic as part of running SugarCRM 7 on-site, as it is a
pre-req. I have a clean Ubuntu 14.04 install with a LAMP stack and Java
installed. I have installed Elasticsearch using apt-get based on the
repository instructions on the Elastic site.

I installed Marvel to see what was happening, and straight away I am seeing
Index growth, even though nothing is happening on the box. I assume it was
some initial indexing as the service get configured, but I left it for a
week and it continued to grow. At this stage no one is using Sugar and
there is no data in it, so it should not be indexing anything.

After a week I have 500,000 documents in the index, and nearly a GB of data
consumed. How could this be?

I have just done a fresh re-install to try again, and it is indexing again.
The index requests on Marvel show a consistent 0.5 per second always.
Where could this be coming from, or how can I fidn out. How can I see
what all the documents allegedly are?

I am still in the process of configuring Sugar, but hesitant to start
loading data and going live until I am comfortable Elastic is running
properly.

Thanks for any suggestions.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

Which index is growing?
Chances are it is the marvel index(es), which is expected.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 June 2014 10:07, Grant Christensen <grant.christensen@supercorp.com.au

wrote:

Hi all, first time Elasticsearch user.

I am using Elastic as part of running SugarCRM 7 on-site, as it is a
pre-req. I have a clean Ubuntu 14.04 install with a LAMP stack and Java
installed. I have installed Elasticsearch using apt-get based on the
repository instructions on the Elastic site.

I installed Marvel to see what was happening, and straight away I am
seeing Index growth, even though nothing is happening on the box. I assume
it was some initial indexing as the service get configured, but I left it
for a week and it continued to grow. At this stage no one is using Sugar
and there is no data in it, so it should not be indexing anything.

After a week I have 500,000 documents in the index, and nearly a GB of
data consumed. How could this be?

I have just done a fresh re-install to try again, and it is indexing
again. The index requests on Marvel show a consistent 0.5 per second
always. Where could this be coming from, or how can I fidn out. How can I
see what all the documents allegedly are?

I am still in the process of configuring Sugar, but hesitant to start
loading data and going live until I am comfortable Elastic is running
properly.

Thanks for any suggestions.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Grant Christensen) #3

Yes that is correct. Looks like a daily Marvel index is growing. Is this the plugin grabbing stats?

Grant Christensen
General Manager - Sales and Product

[More about Supercorp]http://www.supercorp.com.au/

e: Grant.Christensen@supercorp.com.aumailto:%20Grant.Christensen@supercorp.com.au | p: (07) 3850 1200 | f: (07) 3850 1490 | w: http://www.supercorp.com.au

location: Level 4, 16 Marie Street, Milton, QLD 4064, AUSTRALIA | postal: GPO Box 1525, Brisbane, QLD 4001, AUSTRALIA

The information contained in this message is intended only for the use of the individual or entity to which it is addressed. It is confidential and may be legally privileged. If the reader of this message is not the intended recipient, or is not responsible for delivering the message to the intended recipient, then any use, dissemination, further distribution, or reproduction of this message in any form what so ever, is strictly prohibited. If the mail is in error, please notify the sender by return E-mail or telephone and immediately delete your copy of the message. Thank you very much for your assistance.

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Mark Walkom
Sent: Friday, 27 June 2014 10:10 AM
To: elasticsearch@googlegroups.com
Subject: Re: Unsure of growth of index when nothing happening (clean install)

Which index is growing?
Chances are it is the marvel index(es), which is expected.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.commailto:markw@campaignmonitor.com
web: www.campaignmonitor.comhttp://www.campaignmonitor.com

On 27 June 2014 10:07, Grant Christensen <grant.christensen@supercorp.com.aumailto:grant.christensen@supercorp.com.au> wrote:
Hi all, first time Elasticsearch user.

I am using Elastic as part of running SugarCRM 7 on-site, as it is a pre-req. I have a clean Ubuntu 14.04 install with a LAMP stack and Java installed. I have installed Elasticsearch using apt-get based on the repository instructions on the Elastic site.

I installed Marvel to see what was happening, and straight away I am seeing Index growth, even though nothing is happening on the box. I assume it was some initial indexing as the service get configured, but I left it for a week and it continued to grow. At this stage no one is using Sugar and there is no data in it, so it should not be indexing anything.

After a week I have 500,000 documents in the index, and nearly a GB of data consumed. How could this be?

I have just done a fresh re-install to try again, and it is indexing again. The index requests on Marvel show a consistent 0.5 per second always. Where could this be coming from, or how can I fidn out. How can I see what all the documents allegedly are?

I am still in the process of configuring Sugar, but hesitant to start loading data and going live until I am comfortable Elastic is running properly.

Thanks for any suggestions.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/y7GTblbdUDE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B2FD5AED2483AD4590EF405CEEB7B1A901A85B9261%40SCBMAIL01.Supercorp.Local.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

Sure is.

There's more info in the docs
http://www.elasticsearch.org/guide/en/marvel/current/

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 June 2014 10:11, Grant Christensen <Grant.Christensen@supercorp.com.au

wrote:

Yes that is correct. Looks like a daily Marvel index is growing. Is
this the plugin grabbing stats?

Grant Christensen
General Manager - Sales and Product

[image: More about Supercorp] http://www.supercorp.com.au/

e: Grant.Christensen@supercorp.com.au
%20Grant.Christensen@supercorp.com.au | p: (07) 3850 1200 | f: (07) 3850
1490 | w: http://www.supercorp.com.au http://www.supercorp.com.au

location: Level 4, 16 Marie Street, Milton, QLD 4064, AUSTRALIA | postal:
GPO Box 1525, Brisbane, QLD 4001, AUSTRALIA

The information contained in this message is intended only for the use of
the individual or entity to which it is addressed. It is confidential and
may be legally privileged. If the reader of this message is not the
intended recipient, or is not responsible for delivering the message to the
intended recipient, then any use, dissemination, further distribution, or
reproduction of this message in any form what so ever, is strictly
prohibited. If the mail is in error, please notify the sender by return
E-mail or telephone and immediately delete your copy of the message. Thank
you very much for your assistance.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *Mark Walkom
Sent: Friday, 27 June 2014 10:10 AM
To: elasticsearch@googlegroups.com
Subject: Re: Unsure of growth of index when nothing happening (clean
install)

Which index is growing?

Chances are it is the marvel index(es), which is expected.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 June 2014 10:07, Grant Christensen <
grant.christensen@supercorp.com.au> wrote:

Hi all, first time Elasticsearch user.

I am using Elastic as part of running SugarCRM 7 on-site, as it is a
pre-req. I have a clean Ubuntu 14.04 install with a LAMP stack and Java
installed. I have installed Elasticsearch using apt-get based on the
repository instructions on the Elastic site.

I installed Marvel to see what was happening, and straight away I am
seeing Index growth, even though nothing is happening on the box. I assume
it was some initial indexing as the service get configured, but I left it
for a week and it continued to grow. At this stage no one is using Sugar
and there is no data in it, so it should not be indexing anything.

After a week I have 500,000 documents in the index, and nearly a GB of
data consumed. How could this be?

I have just done a fresh re-install to try again, and it is indexing
again. The index requests on Marvel show a consistent 0.5 per second
always. Where could this be coming from, or how can I fidn out. How can I
see what all the documents allegedly are?

I am still in the process of configuring Sugar, but hesitant to start
loading data and going live until I am comfortable Elastic is running
properly.

Thanks for any suggestions.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/y7GTblbdUDE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/B2FD5AED2483AD4590EF405CEEB7B1A901A85B9261%40SCBMAIL01.Supercorp.Local
https://groups.google.com/d/msgid/elasticsearch/B2FD5AED2483AD4590EF405CEEB7B1A901A85B9261%40SCBMAIL01.Supercorp.Local?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Yswk_47nXODKvotr4Tk2RAEPONsymNH7wKwRvK2SO7NA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Boaz Leskes) #5

Grant,

Marvel collects ES stats and put's them in a daily index for historical
reference. You can decide how many days of history you then want to keep.
Curator ( https://github.com/elasticsearch/curator ) is a handy tool to use
in a cronjob to achieve this.

Cheers,
Boaz

On Friday, June 27, 2014 2:36:36 AM UTC+2, Mark Walkom wrote:

Sure is.

There's more info in the docs
http://www.elasticsearch.org/guide/en/marvel/current/

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 June 2014 10:11, Grant Christensen <
Grant.Christensen@supercorp.com.au> wrote:

Yes that is correct. Looks like a daily Marvel index is growing. Is
this the plugin grabbing stats?

Grant Christensen
General Manager - Sales and Product

[image: More about Supercorp] http://www.supercorp.com.au/

e: Grant.Christensen@supercorp.com.au
%20Grant.Christensen@supercorp.com.au | p: (07) 3850 1200 | f: (07) 3850
1490 | w: http://www.supercorp.com.au http://www.supercorp.com.au

location: Level 4, 16 Marie Street, Milton, QLD 4064, AUSTRALIA |
postal: GPO Box 1525, Brisbane, QLD 4001, AUSTRALIA

The information contained in this message is intended only for the use
of the individual or entity to which it is addressed. It is confidential
and may be legally privileged. If the reader of this message is not the
intended recipient, or is not responsible for delivering the message to the
intended recipient, then any use, dissemination, further distribution, or
reproduction of this message in any form what so ever, is strictly
prohibited. If the mail is in error, please notify the sender by return
E-mail or telephone and immediately delete your copy of the message. Thank
you very much for your assistance.

From: elasticsearch@googlegroups.com [mailto:
elasticsearch@googlegroups.com] *On Behalf Of *Mark Walkom
Sent: Friday, 27 June 2014 10:10 AM
To: elasticsearch@googlegroups.com
Subject: Re: Unsure of growth of index when nothing happening (clean
install)

Which index is growing?

Chances are it is the marvel index(es), which is expected.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 27 June 2014 10:07, Grant Christensen <
grant.christensen@supercorp.com.au> wrote:

Hi all, first time Elasticsearch user.

I am using Elastic as part of running SugarCRM 7 on-site, as it is a
pre-req. I have a clean Ubuntu 14.04 install with a LAMP stack and Java
installed. I have installed Elasticsearch using apt-get based on the
repository instructions on the Elastic site.

I installed Marvel to see what was happening, and straight away I am
seeing Index growth, even though nothing is happening on the box. I assume
it was some initial indexing as the service get configured, but I left it
for a week and it continued to grow. At this stage no one is using Sugar
and there is no data in it, so it should not be indexing anything.

After a week I have 500,000 documents in the index, and nearly a GB of
data consumed. How could this be?

I have just done a fresh re-install to try again, and it is indexing
again. The index requests on Marvel show a consistent 0.5 per second
always. Where could this be coming from, or how can I fidn out. How can I
see what all the documents allegedly are?

I am still in the process of configuring Sugar, but hesitant to start
loading data and going live until I am comfortable Elastic is running
properly.

Thanks for any suggestions.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/966e6c58-3e49-4ace-b5cc-758f81d9f24d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/y7GTblbdUDE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEM624aiDk5mykdZZS6_sf68s_8ksRkikHAKhasxYa9VS_dVkw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/B2FD5AED2483AD4590EF405CEEB7B1A901A85B9261%40SCBMAIL01.Supercorp.Local
https://groups.google.com/d/msgid/elasticsearch/B2FD5AED2483AD4590EF405CEEB7B1A901A85B9261%40SCBMAIL01.Supercorp.Local?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ac80a8c-ea3b-4299-a74e-61b868745fb2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6