How to handle searching across all indexes while reindex procedure is running

Elastic Stack Elasticsearch
1

I have a setup that will cause there to be a few hundred indexes, each of
which may need to be rebuilt at any given time. To accomplish this, I'm
creating an alias for each index, which point to an incrementing index
number. For example, the state of the indexes may look like this:

Index -> Alias
indexA -> indexA_v1
indexB -> indexB_v2
indexC -> indexC_v11
indexD -> indexD_v3

An index may need to be rebuilt because the mappings need to change. I
detect when this happens and go through a reindex procedure:

  1. Create a new index (e.g. IndexA_v2) with the new mappings.
  2. Start indexing all the documents into it (could take an hour or two).
  3. Swap the "IndexA" alias to point to IndexA_v2.
  4. Delete IndexA_v1.

I need these to be in separate indexes, not just separate types, because
the mappings can vary widely and potentially include the same field names
but with different mappings. I've learned that mappings should have unique
names within an index, even though types can have separate mappings.

For searches, there are index-specific searches that will just search on
the alias name for the index required (e.g. IndexA). No problem there.

But there is type of search in the app that needs to search across all of
these indexes. But since there could be hundreds of them, adding them to
the search URL seems wrong -- I would think I'd hit some kind of URL limit
at some point:

http://server:9200/indexA,indexB,indexC.....indexZZZ/documenttype/_search

I could use "_all" instead of listing them individually, but with a
specific type that I know only those indexes will have:

http://server:9200/_all/documenttype/_search

When there is no reindex in progress, that works great. But... the "_all"
will include results from any indexes that are in the process of being
rebuilt, meaning that I could get duplicate results from those indexes
during the reindex procedure.

I hope that makes sense. I'm open to a different layout if this isn't the
ideal way to handle it!

Thanks for any help

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Why not creating another alias "index_all" on all indices?
And update it within step 3?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 18 févr. 2015 à 01:23, Dave Reed infinity88@gmail.com a écrit :

I have a setup that will cause there to be a few hundred indexes, each of which may need to be rebuilt at any given time. To accomplish this, I'm creating an alias for each index, which point to an incrementing index number. For example, the state of the indexes may look like this:

Index -> Alias
indexA -> indexA_v1
indexB -> indexB_v2
indexC -> indexC_v11
indexD -> indexD_v3

An index may need to be rebuilt because the mappings need to change. I detect when this happens and go through a reindex procedure:

  1. Create a new index (e.g. IndexA_v2) with the new mappings.
  2. Start indexing all the documents into it (could take an hour or two).
  3. Swap the "IndexA" alias to point to IndexA_v2.
  4. Delete IndexA_v1.

I need these to be in separate indexes, not just separate types, because the mappings can vary widely and potentially include the same field names but with different mappings. I've learned that mappings should have unique names within an index, even though types can have separate mappings.

For searches, there are index-specific searches that will just search on the alias name for the index required (e.g. IndexA). No problem there.

But there is type of search in the app that needs to search across all of these indexes. But since there could be hundreds of them, adding them to the search URL seems wrong -- I would think I'd hit some kind of URL limit at some point:

http://server:9200/indexA,indexB,indexC.....indexZZZ/documenttype/_search

I could use "_all" instead of listing them individually, but with a specific type that I know only those indexes will have:

http://server:9200/_all/documenttype/_search

When there is no reindex in progress, that works great. But... the "_all" will include results from any indexes that are in the process of being rebuilt, meaning that I could get duplicate results from those indexes during the reindex procedure.

I hope that makes sense. I'm open to a different layout if this isn't the ideal way to handle it!

Thanks for any help

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/EEDA5B2E-487B-4EEF-B70F-48731E2634C5%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

3

Ah ha, yes indeed :slight_smile: I knew an index could have multiple aliases, but
didn't connect the dots. That will work perfectly :slight_smile: Thanks!

On Tuesday, February 17, 2015 at 7:39:03 PM UTC-8, David Pilato wrote:

Why not creating another alias "index_all" on all indices?
And update it within step 3?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 18 févr. 2015 à 01:23, Dave Reed <infin...@gmail.com <javascript:>> a
écrit :

I have a setup that will cause there to be a few hundred indexes, each of
which may need to be rebuilt at any given time. To accomplish this, I'm
creating an alias for each index, which point to an incrementing index
number. For example, the state of the indexes may look like this:

Index -> Alias
indexA -> indexA_v1
indexB -> indexB_v2
indexC -> indexC_v11
indexD -> indexD_v3

An index may need to be rebuilt because the mappings need to change. I
detect when this happens and go through a reindex procedure:

  1. Create a new index (e.g. IndexA_v2) with the new mappings.
  2. Start indexing all the documents into it (could take an hour or two).
  3. Swap the "IndexA" alias to point to IndexA_v2.
  4. Delete IndexA_v1.

I need these to be in separate indexes, not just separate types, because
the mappings can vary widely and potentially include the same field names
but with different mappings. I've learned that mappings should have unique
names within an index, even though types can have separate mappings.

For searches, there are index-specific searches that will just search on
the alias name for the index required (e.g. IndexA). No problem there.

But there is type of search in the app that needs to search across all of
these indexes. But since there could be hundreds of them, adding them to
the search URL seems wrong -- I would think I'd hit some kind of URL limit
at some point:

http://server:9200/indexA,indexB,indexC.....indexZZZ/documenttype/_search

I could use "_all" instead of listing them individually, but with a
specific type that I know only those indexes will have:

http://server:9200/_all/documenttype/_search

When there is no reindex in progress, that works great. But... the "_all"
will include results from any indexes that are in the process of being
rebuilt, meaning that I could get duplicate results from those indexes
during the reindex procedure.

I hope that makes sense. I'm open to a different layout if this isn't the
ideal way to handle it!

Thanks for any help

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/7ad24fda-a7ca-4ed8-82ff-70797da1a736%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/968e8714-e7dd-4c3a-8747-a022a147ca16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.