Stability issues due to documention flaw ElasticSearch


(Jasper Siero) #1

We are investigating stability issues we encounter in our ES cluster, and feel that the ES documentation is lacking important information on this topic.

The mailing list seems to fill the gap with the "Document about recommended hardware specs & sharding/index strategy"
http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate...

The information in this conversation looks important (for example shards should not be bigger then the heap size and the heap size should be half of the total memory) but we haven’t found this information in the documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information, what should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB


(Mark Walkom) #2

Your link doesn't work :frowning:

However this sort of info is going to be in the ES (soon to be) book -
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/index.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 21:26, Jasper Siero jasper.siero@target-holding.nl wrote:

We are investigating stability issues we encounter in our ES cluster, and
feel that the ES documentation is lacking import information on this topic.

The mailing list seems to fill the gap with the "Document about recommended
hardware specs & sharding/index strategy"

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate.
..

The information in this conversation looks important (for example shards
should not be bigger then the heap size and the heap size shoud be half of
the total memory) but we haven’t found this information in the
documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information, what
    should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1399634790470-4055631.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jasper Siero) #3

The correct URL is:
http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strategy-td4039398.html

The ES book would be really appreciated. Could you tell me when the release date is planned?

Kind regards,

Jasper Siero


Van: Mark Walkom [via ElasticSearch Users] [ml-node+s115913n4055632h82@n3.nabble.com]
Verzonden: vrijdag 9 mei 2014 13:36
Aan: Jasper Siero
Onderwerp: Re: Stability issues due to documention flaw ElasticSearch

Your link doesn't work :frowning:

However this sort of info is going to be in the ES (soon to be) book - http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/index.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email]<UrlBlockedError.aspx>
web: www.campaignmonitor.comhttp://www.campaignmonitor.com

On 9 May 2014 21:26, Jasper Siero <[hidden email]<UrlBlockedError.aspx>> wrote:
We are investigating stability issues we encounter in our ES cluster, and
feel that the ES documentation is lacking import information on this topic.

The mailing list seems to fill the gap with the "Document about recommended
hardware specs & sharding/index strategy"
http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate...

The information in this conversation looks important (for example shards
should not be bigger then the heap size and the heap size shoud be half of
the total memory) but we haven’t found this information in the
documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information, what
    should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB

--
View this message in context: http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<UrlBlockedError.aspx>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1399634790470-4055631.post%40n3.nabble.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]<UrlBlockedError.aspx>.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.


If you reply to this email, your message will be added to the discussion below:
http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631p4055632.html
To unsubscribe from Stability issues due to documention flaw ElasticSearch, click herehttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4055631&code=amFzcGVyLnNpZXJvQHRhcmdldC1ob2xkaW5nLm5sfDQwNTU2MzF8OTA3NjMyNTM2.
NAMLhttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml


(Mark Walkom) #4

Well O'Reilly has it as March 2014, I'm guessing that is not the fully
edited version - http://shop.oreilly.com/product/0636920028505.do

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 21:46, Jasper Siero jasper.siero@target-holding.nl wrote:

The correct URL is:

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strategy-td4039398.html

The ES book would be really appreciated. Could you tell me when the
release date is planned?

Kind regards,

Jasper Siero


Van: Mark Walkom [via ElasticSearch Users] [ml-node+[hidden email]http://user/SendEmail.jtp?type=node&node=4055633&i=0
]
Verzonden: vrijdag 9 mei 2014 13:36
Aan: Jasper Siero
Onderwerp: Re: Stability issues due to documention flaw ElasticSearch

Your link doesn't work :frowning:

However this sort of info is going to be in the ES (soon to be) book -
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/index.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email] http://UrlBlockedError.aspx
web: www.campaignmonitor.com

On 9 May 2014 21:26, Jasper Siero <[hidden email]http://UrlBlockedError.aspx

wrote:

We are investigating stability issues we encounter in our ES cluster, and
feel that the ES documentation is lacking import information on this
topic.

The mailing list seems to fill the gap with the "Document about
recommended
hardware specs & sharding/index strategy"

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate.
..

The information in this conversation looks important (for example shards
should not be bigger then the heap size and the heap size shoud be half of
the total memory) but we haven’t found this information in the
documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information,
    what
    should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email] http://UrlBlockedError.aspx.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1399634790470-4055631.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email] http://UrlBlockedError.aspx.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.


If you reply to this email, your message will be added to the discussion
below:

http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631p4055632.html
To unsubscribe from Stability issues due to documention flaw
ElasticSearch, click here.
NAMLhttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml


View this message in context: RE: Stability issues due to documention
flaw ElasticSearchhttp://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631p4055633.html
Sent from the ElasticSearch Users mailing list archivehttp://elasticsearch-users.115913.n3.nabble.com/at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/865DC909675A2E40933A72B094A926EA2F5D691D%40mbx5.hosted.exchange-login.nethttps://groups.google.com/d/msgid/elasticsearch/865DC909675A2E40933A72B094A926EA2F5D691D%40mbx5.hosted.exchange-login.net?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bihEH1ugh6aL4BDFcCsRBvXkm08-ZT5qDuYVEPjkRdOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #5

Because each Elasticsearch workload and hardware plus software is
different, you should not expect a serious documentation can always give
"one-size-fits-all" advisory. There are many "rules of thumb" and best
practices, mostly based on individual experience on individual test cases.
The foundation may change in the future, because of other Java version,
because of other hardware specs, or on other Elasticsearch versions etc.

For example, "shards should not be bigger than the heap size" - this might
be useful in situations where massive indexing is going on for that shard
so that segment merging can proceed smoothly, or when you must recover from
broken shards. Beside this, it is also not wrong to have huge shards. If
you have a finalized index, never recover from replica, or never have to
move a shard around, you will not even notice how large your shards are. It
always depends on your requirements.

Jörg

On Fri, May 9, 2014 at 1:26 PM, Jasper Siero <jasper.siero@target-holding.nl

wrote:

We are investigating stability issues we encounter in our ES cluster, and
feel that the ES documentation is lacking import information on this topic.

The mailing list seems to fill the gap with the "Document about recommended
hardware specs & sharding/index strategy"

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate.
..

The information in this conversation looks important (for example shards
should not be bigger then the heap size and the heap size shoud be half of
the total memory) but we haven’t found this information in the
documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information, what
    should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1399634790470-4055631.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGVpWdYuhCD%3DzVoTEJEPxXq-zG8ViFHmm_Qr%3DdAMVJccA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Paige Cook) #6

Here is a post from the Elasticsearch Blog that gives some more context and
status of the book - elasticsearch – the definitive guidehttp://www.elasticsearch.org/blog/elasticsearch-definitive-guide/

On Friday, May 9, 2014 8:03:27 AM UTC-4, Mark Walkom wrote:

Well O'Reilly has it as March 2014, I'm guessing that is not the fully
edited version - http://shop.oreilly.com/product/0636920028505.dohttp://www.google.com/url?q=http%3A%2F%2Fshop.oreilly.com%2Fproduct%2F0636920028505.do&sa=D&sntz=1&usg=AFQjCNHsbG4AB7XXp9MeYfcDgeUKkEfpRw

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 9 May 2014 21:46, Jasper Siero <jasper...@target-holding.nl<javascript:>

wrote:

The correct URL is:

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strategy-td4039398.html

The ES book would be really appreciated. Could you tell me when the
release date is planned?

Kind regards,

Jasper Siero


Van: Mark Walkom [via ElasticSearch Users] [ml-node+[hidden email]http://user/SendEmail.jtp?type=node&node=4055633&i=0
]
Verzonden: vrijdag 9 mei 2014 13:36
Aan: Jasper Siero
Onderwerp: Re: Stability issues due to documention flaw ElasticSearch

Your link doesn't work :frowning:

However this sort of info is going to be in the ES (soon to be) book -
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/index.html

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: [hidden email] http://UrlBlockedError.aspx
web: www.campaignmonitor.com

On 9 May 2014 21:26, Jasper Siero <[hidden email]http://UrlBlockedError.aspx

wrote:

We are investigating stability issues we encounter in our ES cluster, and
feel that the ES documentation is lacking import information on this
topic.

The mailing list seems to fill the gap with the "Document about
recommended
hardware specs & sharding/index strategy"

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate.
..

The information in this conversation looks important (for example shards
should not be bigger then the heap size and the heap size shoud be half
of
the total memory) but we haven’t found this information in the
documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information,
    what
    should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [hidden email] http://UrlBlockedError.aspx.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1399634790470-4055631.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email] http://UrlBlockedError.aspx.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAEM624ZXWHXiaLu1QsAYzwX3oqwYywAsVbg2NnVPig9NKevSyQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.


If you reply to this email, your message will be added to the
discussion below:

http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631p4055632.html
To unsubscribe from Stability issues due to documention flaw
ElasticSearch, click here.
NAMLhttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml


View this message in context: RE: Stability issues due to documention
flaw ElasticSearchhttp://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631p4055633.html
Sent from the ElasticSearch Users mailing list archivehttp://elasticsearch-users.115913.n3.nabble.com/at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/865DC909675A2E40933A72B094A926EA2F5D691D%40mbx5.hosted.exchange-login.nethttps://groups.google.com/d/msgid/elasticsearch/865DC909675A2E40933A72B094A926EA2F5D691D%40mbx5.hosted.exchange-login.net?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3e77eb72-3890-45b1-a8b4-d7736aa64eff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #7

To add to this, you're still on 0.90.X and you should really upgrade as you
will get the benefit of multiple fixes and updates.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 9 May 2014 22:19, joergprante@gmail.com joergprante@gmail.com wrote:

Because each Elasticsearch workload and hardware plus software is
different, you should not expect a serious documentation can always give
"one-size-fits-all" advisory. There are many "rules of thumb" and best
practices, mostly based on individual experience on individual test cases.
The foundation may change in the future, because of other Java version,
because of other hardware specs, or on other Elasticsearch versions etc.

For example, "shards should not be bigger than the heap size" - this might
be useful in situations where massive indexing is going on for that shard
so that segment merging can proceed smoothly, or when you must recover from
broken shards. Beside this, it is also not wrong to have huge shards. If
you have a finalized index, never recover from replica, or never have to
move a shard around, you will not even notice how large your shards are. It
always depends on your requirements.

Jörg

On Fri, May 9, 2014 at 1:26 PM, Jasper Siero <
jasper.siero@target-holding.nl> wrote:

We are investigating stability issues we encounter in our ES cluster, and
feel that the ES documentation is lacking import information on this
topic.

The mailing list seems to fill the gap with the "Document about
recommended
hardware specs & sharding/index strategy"

http://elasticsearch-users.115913.n3.nabble.com/Recommended-Hardware-Specs-amp-Sharding-Index-Strate.
..

The information in this conversation looks important (for example shards
should not be bigger then the heap size and the heap size shoud be half of
the total memory) but we haven’t found this information in the
documentation. We have two questions:

  • Did we overlook this in the official documentation?
  • If the official documentation is lacking this important information,
    what
    should we do to make this taken care of?

Information about our cluster:

4 node Elasticsearch cluster
6 indices in use
Number of active shards 114
version 0.90.3

JVM:
VM name: Java HotSpot(TM) 64-Bit Server VM
VM vendor: Oracle Corporation
VM version: 23.25-b01
Java version: 1.7.0_25

OS:
virtual machine
CentOS 6.5
20 GB memory

CPU:
CPU vendor: Intel
CPU model: Common KVM processor (2099 MHz)
CPU total logical cores: 2
CPU cache: 4kb

Cluster-wide storage size: 140 GB

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Stability-issues-due-to-documention-flaw-ElasticSearch-tp4055631.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1399634790470-4055631.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGVpWdYuhCD%3DzVoTEJEPxXq-zG8ViFHmm_Qr%3DdAMVJccA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAKdsXoGVpWdYuhCD%3DzVoTEJEPxXq-zG8ViFHmm_Qr%3DdAMVJccA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aWi_vQhW4ZbXNoUsx3skMnFpO6MsLJFR4Q%2BJOymeMNdg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #8