Making elasticsearch available ~100% while performing index rebuilding (a specific type) from scratch

Elastic Stack Elasticsearch
1

I have my index created with multiple types, each type represents different
different sets of data, having about 200K documents per type. Number of
types per index could go up to 50 at the max.. At present I have 5 types in
my index.

To make sure that my index is up to date, I have a thread that runs once in
a day to collect any missing documents, deleted documents from the database
and update the index, so that I make sure 100% of the documents are
available to search. To perform this task, I am rebuilding the type from
scratch - by deleting the mapping for the given type and re-creating the
mapping before I start indexing the documents. At present it takes about 30
minutes to re-index some of the types. During this time the search feature
doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding
the index, add a temporary type (say SupplierA_temp for SupplierA), and
once I am done with the rebuilding I could delete the mapping for
SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a
feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Index aliases would be perfect for you:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Essentially, aliases sit on top of your indices and point to one index or
another. Searches then go through the alias instead of the index directly.
In the background, you can create a new index, populate it with data from
your database, then swap the target of the alias from old to new. This
allows you to seamlessly change the index in the background without
interruption to search.

-Zach

On Friday, February 1, 2013 4:28:24 PM UTC-5, Mxims wrote:

I have my index created with multiple types, each type represents
different different sets of data, having about 200K documents per type.
Number of types per index could go up to 50 at the max.. At present I have
5 types in my index.

To make sure that my index is up to date, I have a thread that runs once
in a day to collect any missing documents, deleted documents from the
database and update the index, so that I make sure 100% of the documents
are available to search. To perform this task, I am rebuilding the type
from scratch - by deleting the mapping for the given type and re-creating
the mapping before I start indexing the documents. At present it takes
about 30 minutes to re-index some of the types. During this time the search
feature doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding
the index, add a temporary type (say SupplierA_temp for SupplierA), and
once I am done with the rebuilding I could delete the mapping for
SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a
feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

I don't think you can rename a Type.

But, you can create a new index with the same Type name and use alias to add the new index in addition of the older one and then remove the old type.

Does it help?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 févr. 2013 à 22:28, Mxims renjith.ck@gmail.com a écrit :

I have my index created with multiple types, each type represents different different sets of data, having about 200K documents per type. Number of types per index could go up to 50 at the max.. At present I have 5 types in my index.

To make sure that my index is up to date, I have a thread that runs once in a day to collect any missing documents, deleted documents from the database and update the index, so that I make sure 100% of the documents are available to search. To perform this task, I am rebuilding the type from scratch - by deleting the mapping for the given type and re-creating the mapping before I start indexing the documents. At present it takes about 30 minutes to re-index some of the types. During this time the search feature doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding the index, add a temporary type (say SupplierA_temp for SupplierA), and once I am done with the rebuilding I could delete the mapping for SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Zachary and David thanks a lot!

It looks like alias is my option!!

Let me try it.. I might need some help with creating an alias in Java API.
Anyone got few lines of code to create an alias, swap the index, how to
change the search call to use an alias?

Here is my code to perform search.

SearchRequestBuilder searchBuilder =
client.prepareSearch("supplier").setTypes(supplierCode);
QueryBuilder qb = QueryBuilders.wildcardQuery("name", nameLike);

Thanks

On Friday, February 1, 2013 2:03:23 PM UTC-8, David Pilato wrote:

I don't think you can rename a Type.

But, you can create a new index with the same Type name and use alias to
add the new index in addition of the older one and then remove the old type.

Does it help?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 févr. 2013 à 22:28, Mxims <renji...@gmail.com <javascript:>> a
écrit :

I have my index created with multiple types, each type represents
different different sets of data, having about 200K documents per type.
Number of types per index could go up to 50 at the max.. At present I have
5 types in my index.

To make sure that my index is up to date, I have a thread that runs once
in a day to collect any missing documents, deleted documents from the
database and update the index, so that I make sure 100% of the documents
are available to search. To perform this task, I am rebuilding the type
from scratch - by deleting the mapping for the given type and re-creating
the mapping before I start indexing the documents. At present it takes
about 30 minutes to re-index some of the types. During this time the search
feature doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding
the index, add a temporary type (say SupplierA_temp for SupplierA), and
once I am done with the rebuilding I could delete the mapping for
SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a
feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi

try this one if you have not found it yet:

Cheers,
Alex

Am Samstag, 2. Februar 2013 01:27:31 UTC+1 schrieb Mxims:

Zachary and David thanks a lot!

It looks like alias is my option!!

Let me try it.. I might need some help with creating an alias in Java API.
Anyone got few lines of code to create an alias, swap the index, how to
change the search call to use an alias?

Here is my code to perform search.

SearchRequestBuilder searchBuilder =
client.prepareSearch("supplier").setTypes(supplierCode);
QueryBuilder qb = QueryBuilders.wildcardQuery("name", nameLike);

Thanks

On Friday, February 1, 2013 2:03:23 PM UTC-8, David Pilato wrote:

I don't think you can rename a Type.

But, you can create a new index with the same Type name and use alias to
add the new index in addition of the older one and then remove the old type.

Does it help?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 févr. 2013 à 22:28, Mxims renji...@gmail.com a écrit :

I have my index created with multiple types, each type represents
different different sets of data, having about 200K documents per type.
Number of types per index could go up to 50 at the max.. At present I have
5 types in my index.

To make sure that my index is up to date, I have a thread that runs once
in a day to collect any missing documents, deleted documents from the
database and update the index, so that I make sure 100% of the documents
are available to search. To perform this task, I am rebuilding the type
from scratch - by deleting the mapping for the given type and re-creating
the mapping before I start indexing the documents. At present it takes
about 30 minutes to re-index some of the types. During this time the search
feature doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding
the index, add a temporary type (say SupplierA_temp for SupplierA), and
once I am done with the rebuilding I could delete the mapping for
SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a
feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

This works really well, btw. Alias swapping is atomic. In out case the
only thing our users might notice is a little backwards time warp effect
when we switch the alias from the old index to the new one. But they only
get that because we're constantly updating the index with new stuff so we
have to run a catch up script after the swap. It looks like the swap would
be your catch up mechanism so you are all good there.

Nik

On Fri, Nov 8, 2013 at 5:24 AM, Alexander Kley alexander.kley@gmail.comwrote:

Hi

try this one if you have not found it yet:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Cheers,
Alex

Am Samstag, 2. Februar 2013 01:27:31 UTC+1 schrieb Mxims:

Zachary and David thanks a lot!

It looks like alias is my option!!

Let me try it.. I might need some help with creating an alias in Java
API. Anyone got few lines of code to create an alias, swap the index, how
to change the search call to use an alias?

Here is my code to perform search.

SearchRequestBuilder searchBuilder = client.prepareSearch("
supplier").setTypes(supplierCode);
QueryBuilder qb = QueryBuilders.wildcardQuery("name", nameLike);

Thanks

On Friday, February 1, 2013 2:03:23 PM UTC-8, David Pilato wrote:

I don't think you can rename a Type.

But, you can create a new index with the same Type name and use alias to
add the new index in addition of the older one and then remove the old type.

Does it help?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 1 févr. 2013 à 22:28, Mxims renji...@gmail.com a écrit :

I have my index created with multiple types, each type represents
different different sets of data, having about 200K documents per type.
Number of types per index could go up to 50 at the max.. At present I have
5 types in my index.

To make sure that my index is up to date, I have a thread that runs once
in a day to collect any missing documents, deleted documents from the
database and update the index, so that I make sure 100% of the documents
are available to search. To perform this task, I am rebuilding the type
from scratch - by deleting the mapping for the given type and re-creating
the mapping before I start indexing the documents. At present it takes
about 30 minutes to re-index some of the types. During this time the search
feature doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding
the index, add a temporary type (say SupplierA_temp for SupplierA), and
once I am done with the rebuilding I could delete the mapping for
SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a
feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

I've written a python script which will swap your data effortlessly

On Friday, 1 February 2013 21:28:24 UTC, Mxims wrote:

I have my index created with multiple types, each type represents
different different sets of data, having about 200K documents per type.
Number of types per index could go up to 50 at the max.. At present I have
5 types in my index.

To make sure that my index is up to date, I have a thread that runs once
in a day to collect any missing documents, deleted documents from the
database and update the index, so that I make sure 100% of the documents
are available to search. To perform this task, I am rebuilding the type
from scratch - by deleting the mapping for the given type and re-creating
the mapping before I start indexing the documents. At present it takes
about 30 minutes to re-index some of the types. During this time the search
feature doesn't work since my type in the index is almost empty!

To overcome this, here is the idea I have in my mind - While rebuilding
the index, add a temporary type (say SupplierA_temp for SupplierA), and
once I am done with the rebuilding I could delete the mapping for
SupplierA, and then renaming SupplierA-temp to SupplierA. Is that a
feasible solution? or is there a better solution?

Thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ea3ce388-dac1-429b-8cad-85f708e937d8%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.