Windows Elasticsearch cluster performance tuning


(Eric Brandes) #1

Hey all, I have a 3 node Elasticsearch 1.0.1 cluster running on Windows
Server 2012 (in Azure). There's about 20 million documents that take up a
total of 40GB (including replicas). There's about 400 indexes in total,
with some having millions of documents and some having just a few. Each
index is set to have 3 shards and 1 replica. The main cluster is running
on three 4 core machines with 7GB of ram. The min/max JVM heap size is
set to 4GB.

The primary use case for this cluster is faceting/aggregations over the
documents. There's almost no full text searching, so everything is pretty
much based on exact values (which are stored but not analyzed at index time)

When doing some term facets on a few of these indexes (the biggest one
contains about 8 million documents) I'm seeing really long response times
(> 5 sec). There are potentially thousands of distinct values for the term
I'm faceting on, but I would have still expected faster performance.

So my goal is to speed up these queries to get the responses sub second if
possible. To that end I had some questions:

  1. Would switching to Linux give me better performance in general?
  2. I could collapse almost all of these 400 indexes in to a single big
    index and use aliases + filters instead. Would this be advisable?
  3. Would mucking with the field data cache yield any better results?

If I can add any more data to this discussion please let me know!
Thanks!
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb5fb6bf-be2c-4d5f-b73a-edc1ef5813f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #2

IMHO 800 shards per node is far too much. And with only 4gb of memory...

I guess you have lot of GC or you forget to disable SWAP.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 18:08, Eric Brandes eric.brandes@gmail.com a écrit :

Hey all, I have a 3 node Elasticsearch 1.0.1 cluster running on Windows Server 2012 (in Azure). There's about 20 million documents that take up a total of 40GB (including replicas). There's about 400 indexes in total, with some having millions of documents and some having just a few. Each index is set to have 3 shards and 1 replica. The main cluster is running on three 4 core machines with 7GB of ram. The min/max JVM heap size is set to 4GB.

The primary use case for this cluster is faceting/aggregations over the documents. There's almost no full text searching, so everything is pretty much based on exact values (which are stored but not analyzed at index time)

When doing some term facets on a few of these indexes (the biggest one contains about 8 million documents) I'm seeing really long response times (> 5 sec). There are potentially thousands of distinct values for the term I'm faceting on, but I would have still expected faster performance.

So my goal is to speed up these queries to get the responses sub second if possible. To that end I had some questions:

  1. Would switching to Linux give me better performance in general?
  2. I could collapse almost all of these 400 indexes in to a single big index and use aliases + filters instead. Would this be advisable?
  3. Would mucking with the field data cache yield any better results?

If I can add any more data to this discussion please let me know!
Thanks!
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb5fb6bf-be2c-4d5f-b73a-edc1ef5813f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C6157A06-390B-45C0-8425-3723F37D3766%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #3

Forget to say that you should extra large instances and not large.
With larges, you could suffer from noisy neighbors.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 19:54, David Pilato david@pilato.fr a écrit :

IMHO 800 shards per node is far too much. And with only 4gb of memory...

I guess you have lot of GC or you forget to disable SWAP.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 18:08, Eric Brandes eric.brandes@gmail.com a écrit :

Hey all, I have a 3 node Elasticsearch 1.0.1 cluster running on Windows Server 2012 (in Azure). There's about 20 million documents that take up a total of 40GB (including replicas). There's about 400 indexes in total, with some having millions of documents and some having just a few. Each index is set to have 3 shards and 1 replica. The main cluster is running on three 4 core machines with 7GB of ram. The min/max JVM heap size is set to 4GB.

The primary use case for this cluster is faceting/aggregations over the documents. There's almost no full text searching, so everything is pretty much based on exact values (which are stored but not analyzed at index time)

When doing some term facets on a few of these indexes (the biggest one contains about 8 million documents) I'm seeing really long response times (> 5 sec). There are potentially thousands of distinct values for the term I'm faceting on, but I would have still expected faster performance.

So my goal is to speed up these queries to get the responses sub second if possible. To that end I had some questions:

  1. Would switching to Linux give me better performance in general?
  2. I could collapse almost all of these 400 indexes in to a single big index and use aliases + filters instead. Would this be advisable?
  3. Would mucking with the field data cache yield any better results?

If I can add any more data to this discussion please let me know!
Thanks!
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb5fb6bf-be2c-4d5f-b73a-edc1ef5813f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C6157A06-390B-45C0-8425-3723F37D3766%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0AB6FD4E-0C22-4256-8E29-2F40C06E793E%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(Eric Brandes) #4

Interesting - so in general would you recommend consolidating all 400
indexes in to a single index and using aliases/filters to address them?
(they're currently broken out by user, and all operations are scoped to a
specific user)

If I were to consolidate to a single index, how many shards would be
recommended?

On Sunday, March 23, 2014 2:00:18 PM UTC-5, David Pilato wrote:

Forget to say that you should extra large instances and not large.
With larges, you could suffer from noisy neighbors.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 19:54, David Pilato <da...@pilato.fr <javascript:>> a
écrit :

IMHO 800 shards per node is far too much. And with only 4gb of memory...

I guess you have lot of GC or you forget to disable SWAP.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 18:08, Eric Brandes <eric.b...@gmail.com <javascript:>>
a écrit :

Hey all, I have a 3 node Elasticsearch 1.0.1 cluster running on Windows
Server 2012 (in Azure). There's about 20 million documents that take up a
total of 40GB (including replicas). There's about 400 indexes in total,
with some having millions of documents and some having just a few. Each
index is set to have 3 shards and 1 replica. The main cluster is running
on three 4 core machines with 7GB of ram. The min/max JVM heap size is
set to 4GB.

The primary use case for this cluster is faceting/aggregations over the
documents. There's almost no full text searching, so everything is pretty
much based on exact values (which are stored but not analyzed at index time)

When doing some term facets on a few of these indexes (the biggest one
contains about 8 million documents) I'm seeing really long response times
(> 5 sec). There are potentially thousands of distinct values for the term
I'm faceting on, but I would have still expected faster performance.

So my goal is to speed up these queries to get the responses sub second if
possible. To that end I had some questions:

  1. Would switching to Linux give me better performance in general?
  2. I could collapse almost all of these 400 indexes in to a single big
    index and use aliases + filters instead. Would this be advisable?
  3. Would mucking with the field data cache yield any better results?

If I can add any more data to this discussion please let me know!
Thanks!
Eric

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eb5fb6bf-be2c-4d5f-b73a-edc1ef5813f1%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/eb5fb6bf-be2c-4d5f-b73a-edc1ef5813f1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/C6157A06-390B-45C0-8425-3723F37D3766%40pilato.frhttps://groups.google.com/d/msgid/elasticsearch/C6157A06-390B-45C0-8425-3723F37D3766%40pilato.fr?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04ea8acb-a62f-4232-a483-bdde916c48c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(David Pilato) #5

Yes. Hard to say. You need to test how much docs a single shard could hold. It depends on your use case.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 20:11, Eric Brandes eric.brandes@gmail.com a écrit :

Interesting - so in general would you recommend consolidating all 400 indexes in to a single index and using aliases/filters to address them? (they're currently broken out by user, and all operations are scoped to a specific user)

If I were to consolidate to a single index, how many shards would be recommended?

On Sunday, March 23, 2014 2:00:18 PM UTC-5, David Pilato wrote:
Forget to say that you should extra large instances and not large.
With larges, you could suffer from noisy neighbors.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 19:54, David Pilato da...@pilato.fr a écrit :

IMHO 800 shards per node is far too much. And with only 4gb of memory...

I guess you have lot of GC or you forget to disable SWAP.

My 2 cents.

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 23 mars 2014 à 18:08, Eric Brandes eric.b...@gmail.com a écrit :

Hey all, I have a 3 node Elasticsearch 1.0.1 cluster running on Windows Server 2012 (in Azure). There's about 20 million documents that take up a total of 40GB (including replicas). There's about 400 indexes in total, with some having millions of documents and some having just a few. Each index is set to have 3 shards and 1 replica. The main cluster is running on three 4 core machines with 7GB of ram. The min/max JVM heap size is set to 4GB.

The primary use case for this cluster is faceting/aggregations over the documents. There's almost no full text searching, so everything is pretty much based on exact values (which are stored but not analyzed at index time)

When doing some term facets on a few of these indexes (the biggest one contains about 8 million documents) I'm seeing really long response times (> 5 sec). There are potentially thousands of distinct values for the term I'm faceting on, but I would have still expected faster performance.

So my goal is to speed up these queries to get the responses sub second if possible. To that end I had some questions:

  1. Would switching to Linux give me better performance in general?
  2. I could collapse almost all of these 400 indexes in to a single big index and use aliases + filters instead. Would this be advisable?
  3. Would mucking with the field data cache yield any better results?

If I can add any more data to this discussion please let me know!
Thanks!
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eb5fb6bf-be2c-4d5f-b73a-edc1ef5813f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/C6157A06-390B-45C0-8425-3723F37D3766%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/04ea8acb-a62f-4232-a483-bdde916c48c2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B736C9EB-1617-483C-8900-A5F7CDDAD8DA%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.


(system) #6