# Performance on indices for each language

**URL:** <https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222>\
**Category:** Elasticsearch\
**Created:** [June 2, 2013, 9:06am UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222 "2013-06-02T09:06:16Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Felipe\_Hummel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felipe_hummel/32/1263_2.png) [@Felipe\_Hummel](https://discuss.elastic.co/u/Felipe_Hummel)\
**Post date:** [June 2, 2013, 9:06am UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222/1 "2013-06-02T09:06:16Z")

</div>

Hi, we currently one unique index in production which has around 50M  
documents. They are, mainly, from three different languages and also have a  
created\_at field associated with it.

Our common search use case is to search documents of any language but we  
allow users to search only one specific language. Also, in terms of time  
intervals, the common use case is to search the entire dataset, but we also  
have a use case where we only need to search documents of the last ~1 year.

We currently have some queries paying the price of searching in a 50M  
index, when they would only need to search in \< 10M documents. Filters can  
help with this, but they only help skipping the scoring of a document, the  
query processing still needs to go through the whole gigantic posting list,  
that is potentially on disk. (a posting list for "the" in Spanish is really  
tiny compared to the one in English, for example).

My questions are regarding performance. Our main concern, now, is query  
latency. Our system uses many queries with many _OR_ clauses which easily  
makes query latency a pain, depending on the terms it can be up to dozens  
of seconds.

- First, if I search in multiple indices, will the search on them be done  
in "parallel"? For example, I have "alias1" over two indexes: "index1" and  
"index2". When searching in "alias1", the search in "index1" will occur in  
parallel to the search in "index2", or they will be executed in sequence?
- Second, what are the implications of "cutting" the dataset into 3  
indices, one for each language? What will be the performance _difference  
between searching 3 indices and searching 1 index with all documents_?
- Third, separating indices by language and then searching all indices  
together would mess up the scoring, right? IDFs for words can greatly vary between  
languages (would need to change search type?)
- Finally, it would be a good idea to "cut" the index in time intervals?  
(like an index for each year worth of documents, for each language).

This all assumes the same number of machines/shards/replicas. We currently  
have 16 shards in 8 (m1.large) EC2 instances.

Thanks!

Felipe Hummel

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Andrew\_Gaydenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_gaydenko/32/914_2.png) [@Andrew\_Gaydenko](https://discuss.elastic.co/u/Andrew_Gaydenko)\
**Post date:** [June 2, 2013, 9:18am UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222/2 "2013-06-02T09:18:30Z")

</div>

Do you use _filtered_ queries?  
[http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query/](http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query/)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Felipe\_Hummel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felipe_hummel/32/1263_2.png) [@Felipe\_Hummel](https://discuss.elastic.co/u/Felipe_Hummel)\
**Post date:** [June 2, 2013, 4:27pm UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222/3 "2013-06-02T16:27:17Z")

</div>

Not actually, we normally use the request-level filters  
([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/filter/)).  
We only use filtered when we need some facet to be calculated considering  
the filters.

I assume there's no performance difference.

On Sunday, June 2, 2013 5:18:30 AM UTC-4, Andrew Gaydenko wrote:

> Do you use _filtered_ queries?  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query/)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Felipe\_Hummel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felipe_hummel/32/1263_2.png) [@Felipe\_Hummel](https://discuss.elastic.co/u/Felipe_Hummel)\
**Post date:** [June 4, 2013, 8:06pm UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222/4 "2013-06-04T20:06:48Z")

</div>

Anyone has an opinion?

On Sunday, June 2, 2013 12:27:17 PM UTC-4, Felipe Hummel wrote:

> Not actually, we normally use the request-level filters (  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/filter/)).  
> We only use filtered when we need some facet to be calculated considering  
> the filters.
> 
> I assume there's no performance difference.
> 
> On Sunday, June 2, 2013 5:18:30 AM UTC-4, Andrew Gaydenko wrote:
> 
> > Do you use _filtered_ queries?  
> > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/filtered-query/)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\
**Post date:** [June 5, 2013, 8:13am UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222/5 "2013-06-05T08:13:02Z")

</div>

Hi,

One element that people often don't talk about when talking about searching  
documents in multiple languages is the UI/UX. What does that need to look  
like and how much flexibility do you have there? I understand some queries  
need to search all languages, but that doesn't necessarily mean the UI  
needs to show a single result set. If the presentation layer allows  
separation by language, I would go with index-per-language model, which is  
cleaner and simpler.

The second part of your email is about searching all content vs. a subset  
of content depending on the user time range selection, or something along  
those lines. For this you could consider having multiple indices for  
different time frames and the search client that knows which index holds  
documents in which time range and issues queries only to the relevant  
indices. Alternatively, this might be doable with routing on a field that  
contains a date or a part of it.

## Otis

Solr & Elasticsearch Support - [http://sematext.com/](http://sematext.com/)  
Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html)

On Sunday, June 2, 2013 5:06:16 AM UTC-4, Felipe Hummel wrote:

> Hi, we currently one unique index in production which has around 50M  
> documents. They are, mainly, from three different languages and also have a  
> created\_at field associated with it.
> 
> Our common search use case is to search documents of any language but we  
> allow users to search only one specific language. Also, in terms of time  
> intervals, the common use case is to search the entire dataset, but we also  
> have a use case where we only need to search documents of the last ~1 year.
> 
> We currently have some queries paying the price of searching in a 50M  
> index, when they would only need to search in \< 10M documents. Filters can  
> help with this, but they only help skipping the scoring of a document, the  
> query processing still needs to go through the whole gigantic posting list,  
> that is potentially on disk. (a posting list for "the" in Spanish is really  
> tiny compared to the one in English, for example).
> 
> My questions are regarding performance. Our main concern, now, is query  
> latency. Our system uses many queries with many _OR_ clauses which easily  
> makes query latency a pain, depending on the terms it can be up to dozens  
> of seconds.
> 
> - First, if I search in multiple indices, will the search on them be done  
> in "parallel"? For example, I have "alias1" over two indexes: "index1"  
> and "index2". When searching in "alias1", the search in "index1" will occur  
> in parallel to the search in "index2", or they will be executed in sequence?
> - Second, what are the implications of "cutting" the dataset into 3  
> indices, one for each language? What will be the performance _difference  
> between searching 3 indices and searching 1 index with all documents_?
> - Third, separating indices by language and then searching all indices  
> together would mess up the scoring, right? IDFs for words can greatly vary between  
> languages (would need to change search type?)
> - Finally, it would be a good idea to "cut" the index in time intervals?  
> (like an index for each year worth of documents, for each language).
> 
> This all assumes the same number of machines/shards/replicas. We currently  
> have 16 shards in 8 (m1.large) EC2 instances.
> 
> Thanks!
> 
> Felipe Hummel

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:32am UTC](https://discuss.elastic.co/t/performance-on-indices-for-each-language/12222/6 "2017-07-06T02:32:50Z")

</div>