Limit search on number of indices

Elastic Stack Elasticsearch
1

Hi there,

Is there a (hard) limit on the number of indices involved in a search.

My ES design is really simple: one index per day. When I search on 1 week
of data, I generate the dates of the last 7 days and perform a search on
them. E.g. 2012-12-20,2012-12-21,...,2012-12-27

However when I try to search one month of data it seems that something goes
wrong. Is this a limit of indices to be searched at once?

Best regards,

Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

--

2

The main issue I can see is the "too many open files" problem.
Do you see that in logs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 27 déc. 2012 à 12:58, Robin Verlangen robin@us2.nl a écrit :

Hi there,

Is there a (hard) limit on the number of indices involved in a search.

My ES design is really simple: one index per day. When I search on 1 week of data, I generate the dates of the last 7 days and perform a search on them. E.g. 2012-12-20,2012-12-21,...,2012-12-27

However when I try to search one month of data it seems that something goes wrong. Is this a limit of indices to be searched at once?

Best regards,

Robin Verlangen
Software engineer

W http://www.robinverlangen.nl
E robin@us2.nl

Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.

--

3

So there is no limit on the amount of indices being searched, apart from
the open file limit?

Best regards,

Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

On Thu, Dec 27, 2012 at 1:21 PM, David Pilato david@pilato.fr wrote:

The main issue I can see is the "too many open files" problem.
Do you see that in logs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 27 déc. 2012 à 12:58, Robin Verlangen robin@us2.nl a écrit :

Hi there,

Is there a (hard) limit on the number of indices involved in a search.

My ES design is really simple: one index per day. When I search on 1 week
of data, I generate the dates of the last 7 days and perform a search on
them. E.g. 2012-12-20,2012-12-21,...,2012-12-27

However when I try to search one month of data it seems that something
goes wrong. Is this a limit of indices to be searched at once?

Best regards,

Robin Verlangen
Software engineer
*
*
W http://www.robinverlangen.nl
E robin@us2.nl

http://goo.gl/Lt7BC

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.

--

--

--

4

Hey Robin,

Perhaps you could elaborate a bit more your issue. Are you using aliases? Are
you searching on the whole cluster?
31 indices should not be an issue AFAIK.

BTW, here's what Igor wrote some months ago on stackoverflow
http://stackoverflow.com/questions/12572234/elasticsearch-separate-index-per-user/12598894#12598894
:

"I can also add that having a lot of indices will slow down cluster restarts and
repeating user data will make modification and deletion of user records more
complicated that it needs to be. In other words, you can have a lot of indices,
but in your case it's, probably, a bad idea."

David.

Le 27 décembre 2012 à 16:25, Robin Verlangen robin@us2.nl a écrit :

So there is no limit on the amount of indices being searched, apart from the
open file limit?

Best regards,

Robin Verlangen
Software engineer

W http://www.robinverlangen.nl http://www.robinverlangen.nl
E robin@us2.nl mailto:robin@us2.nl
http://goo.gl/Lt7BC

Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that the
information remains the property of the sender. You must not use, disclose,
distribute, copy, print or rely on this e-mail. If you have received this
message in error, please contact the sender immediately and irrevocably delete
this message and any copies.

On Thu, Dec 27, 2012 at 1:21 PM, David Pilato <david@pilato.fr
mailto:david@pilato.fr > wrote:

The main issue I can see is the "too many open files" problem.
Do you see that in logs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 27 déc. 2012 à 12:58, Robin Verlangen < robin@us2.nl
mailto:robin@us2.nl > a écrit :

   > > >        Hi there,

   Is there a (hard) limit on the number of indices involved in a

search.

   My ES design is really simple: one index per day. When I search on

1 week of data, I generate the dates of the last 7 days and perform a
search on them. E.g. 2012-12-20,2012-12-21,...,2012-12-27

   However when I try to search one month of data it seems that

something goes wrong. Is this a limit of indices to be searched at once?

   Best regards,

   Robin Verlangen
   Software engineer

   W http://www.robinverlangen.nl <http://www.robinverlangen.nl>
   E robin@us2.nl <mailto:robin@us2.nl>
     <http://goo.gl/Lt7BC>

   Disclaimer: The information contained in this message and

attachments is intended solely for the attention and use of the named
addressee and may be confidential. If you are not the intended recipient,
you are reminded that the information remains the property of the sender.
You must not use, disclose, distribute, copy, print or rely on this
e-mail. If you have received this message in error, please contact the
sender immediately and irrevocably delete this message and any copies.

   --

--

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--