I'm a bit stuck trying to figure out the best way to model my indices for
aggregations. I'm currently storing article hits in indices that roll over
each month. Each index tends to have around 60M records. However, I have
two concerns:
In the future I expect the number of indices will grow into the
hundreds. If I'm trying to aggregate the total number of hits or the hits
per month of an article across the many indices, will the query end up
getting very slow since it has to aggregate across them all? Would it be
better to store all the hits for an article in the same index and use a new
index for blocks of article IDs instead of a new index per month to make
the index predictable for a certain article?
What about when I want to see what the top 10 articles of all time are?
This would require doing an aggregation of all articles across all indices,
right? How slow will that get when there are hundreds of indices with 60M+
records per index? Would it be better to store a hit counter on the article
record itself that gets updated occasionally?
Is there a better way to model the indices that would accommodate both of
these use cases?
For 1., storing all hits for an article in the same index would not help
performance. ALso note that you mentioned numbers of indices but what
really matters to Elasticsearch is the total number of primary shards.
Having 60 indices with 1 shard is much better to Elasticsearch than one
index with 1000 shards.
Would it be better to store a hit counter on the article record itself
that gets updated occasionally?
I think this is an option that you should consider indeed. Running a query
with a sort is much more efficient than running a terms aggregation to
compute the article that has most hits.
How can I learn more about the overhead of shards? As I understand it, the
more documents there are per shard, the slower query performance is. If
there is only one shard per index then there is no parallelization of the
query. Right now I have an average of 60 million records per month going
into an index with the default of 5 shards and I'm creating a new index
each month. That means there's an average of about 12 million records per
shard. Perhaps I should decrease the number of shards, but I'm concerned
about query performance if I get too many documents per shard. With this
scheme that would leave me with 60 new shards per year. Of course, I have 4
data nodes and I'm planning to expand this further so they don't all have
to be on one node.
I guess what I'd really like to understand then is, what's a reasonable
number of shards per node? What's the overhead of a new active primary
shard? I can continue to spin up servers and scale horizontally adding new
nodes to handle the shards, I just need a good way to gauge how to tell
when to add a node to the cluster. Also, I imagine if I'm searching across
all indices it will wreck query performance just because at some point
there will be so many that the parallelization itself introduces too much
overhead. Hence, the idea of limiting article hits to a subset of indices
based on both article id and date to prevent queries from having to touch
everything as the data grows.
On Thursday, February 26, 2015 at 4:31:44 PM UTC-7, Adrien Grand wrote:
For 1., storing all hits for an article in the same index would not help
performance. ALso note that you mentioned numbers of indices but what
really matters to Elasticsearch is the total number of primary shards.
Having 60 indices with 1 shard is much better to Elasticsearch than one
index with 1000 shards.
Would it be better to store a hit counter on the article record itself
that gets updated occasionally?
I think this is an option that you should consider indeed. Running a query
with a sort is much more efficient than running a terms aggregation to
compute the article that has most hits.
The overhead of shards boils down to the fact that shards are Lucene
indices. There is not exact number for the right number of shards per node
although 3 is certainly OK and 100 probably too much. It's true that today
shards are processed in a single thread, so having fewer larger shards
might increase latency a bit (but it does not hurt throughput however).
The chapter about capacity planning from the book might be helpful too:
How can I learn more about the overhead of shards? As I understand it, the
more documents there are per shard, the slower query performance is. If
there is only one shard per index then there is no parallelization of the
query. Right now I have an average of 60 million records per month going
into an index with the default of 5 shards and I'm creating a new index
each month. That means there's an average of about 12 million records per
shard. Perhaps I should decrease the number of shards, but I'm concerned
about query performance if I get too many documents per shard. With this
scheme that would leave me with 60 new shards per year. Of course, I have 4
data nodes and I'm planning to expand this further so they don't all have
to be on one node.
I guess what I'd really like to understand then is, what's a reasonable
number of shards per node? What's the overhead of a new active primary
shard? I can continue to spin up servers and scale horizontally adding new
nodes to handle the shards, I just need a good way to gauge how to tell
when to add a node to the cluster. Also, I imagine if I'm searching across
all indices it will wreck query performance just because at some point
there will be so many that the parallelization itself introduces too much
overhead. Hence, the idea of limiting article hits to a subset of indices
based on both article id and date to prevent queries from having to touch
everything as the data grows.
On Thursday, February 26, 2015 at 4:31:44 PM UTC-7, Adrien Grand wrote:
For 1., storing all hits for an article in the same index would not help
performance. ALso note that you mentioned numbers of indices but what
really matters to Elasticsearch is the total number of primary shards.
Having 60 indices with 1 shard is much better to Elasticsearch than one
index with 1000 shards.
Would it be better to store a hit counter on the article record itself
that gets updated occasionally?
I think this is an option that you should consider indeed. Running a
query with a sort is much more efficient than running a terms aggregation
to compute the article that has most hits.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.