# Index "bootstrapping" **URL:** \ **Category:** Elasticsearch\ **Created:** [November 8, 2010, 11:48pm UTC](https://discuss.elastic.co/t/index-bootstrapping/3537 "2010-11-08T23:48:06Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ryan\_Crumley](https://avatars.discourse-cdn.com/v4/letter/r/b9e5f3/32.png) [@Ryan\_Crumley](https://discuss.elastic.co/u/Ryan_Crumley)\ **Post date:** [November 8, 2010, 11:48pm UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/1 "2010-11-08T23:48:06Z")
All, I am integrating Elastic Search into a new java/spring/hibernate/ wicket application. I want to keep this application as standalone as possible so I opted to integrate ES into the web application itself (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has been working great with the exception of "bootstrapping" the index at startup. The current bootstrap flow looks like this: 1. Web application is started. 2. Search service tries to run a count() query to determine if a full index is needed. I run into the first issue here when the index has not completed loading from the local gateway. This causes the count query fail. If I wait and retry the count query it eventually completes with the expected answer and I can decide if I need to reindex the content. In the case the index does not exist I try to create a new index by executing a "create index" call: client.admin().indices().prepareCreate(INDEX\_NAME).execute(); Then immediately I want to customize a mapping but sometimes a "put mapping" call fails because the index is not fully created. Again if I wait long enough after the "create index" call the "put mapping" will succeed however it seems like there should be a better way. Any suggestions? The data set I am using is around 200-500 small documents, with 3-4 terms. I don't anticipate needing a second ES server any time soon so ideally any solution would not involve a second ES node. I have tried using the "local" and "file" gateway with similar results. Since the current dataset is small if I had to index each time the application started (no persistence of the index) that would be OK however I still have the "create index" / "put mapping" race condition. My current ES configuration is very simple: gateway.type=local gateway.recover\_after\_nodes=0 gateway.recover\_after\_time=0s Thanks in advance, Ryan ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [November 9, 2010, 12:30am UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/2 "2010-11-09T00:30:44Z")
Hi, there is a cluster health API that you can use to check if new index is already allocated and ready for use. [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) Regards, Lukas On Tue, Nov 9, 2010 at 12:48 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > All, > > I am integrating Elastic Search into a new java/spring/hibernate/ > wicket application. I want to keep this application as standalone as > possible so I opted to integrate ES into the web application itself > (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has > been working great with the exception of "bootstrapping" the index at > startup. The current bootstrap flow looks like this: > > 1. Web application is started. > 2. Search service tries to run a count() query to determine if a full > index is needed. > > I run into the first issue here when the index has not completed > loading from the local gateway. This causes the count query fail. If I > wait and retry the count query it eventually completes with the > expected answer and I can decide if I need to reindex the content. > > In the case the index does not exist I try to create a new index by > executing a "create index" call: > > client.admin().indices().prepareCreate(INDEX\_NAME).execute(); > > Then immediately I want to customize a mapping but sometimes a "put > mapping" call fails because the index is not fully created. Again if I > wait long enough after the "create index" call the "put mapping" will > succeed however it seems like there should be a better way. Any > suggestions? > > The data set I am using is around 200-500 small documents, with 3-4 > terms. I don't anticipate needing a second ES server any time soon so > ideally any solution would not involve a second ES node. I have tried > using the "local" and "file" gateway with similar results. Since the > current dataset is small if I had to index each time the application > started (no persistence of the index) that would be OK however I still > have the "create index" / "put mapping" race condition. > > My current ES configuration is very simple: > > gateway.type=local > gateway.recover\_after\_nodes=0 > gateway.recover\_after\_time=0s > > Thanks in advance, > > Ryan ---
**Author:** ![Ryan\_Crumley](https://avatars.discourse-cdn.com/v4/letter/r/b9e5f3/32.png) [@Ryan\_Crumley](https://discuss.elastic.co/u/Ryan_Crumley)\ **Post date:** [November 9, 2010, 12:36am UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/3 "2010-11-09T00:36:45Z")
Thanks I will take a look. Is there no better way to structure this than polling between API calls? How are people managing their custom type mappings? Ryan On Mon, Nov 8, 2010 at 6:30 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Hi, > > there is a cluster health API that you can use to check if new index is > already allocated and ready for use. > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) > Regards, > Lukas > > On Tue, Nov 9, 2010 at 12:48 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > > > All, > > > > I am integrating Elastic Search into a new java/spring/hibernate/ > > wicket application. I want to keep this application as standalone as > > possible so I opted to integrate ES into the web application itself > > (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has > > been working great with the exception of "bootstrapping" the index at > > startup. The current bootstrap flow looks like this: > > > > 1. Web application is started. > > 2. Search service tries to run a count() query to determine if a full > > index is needed. > > > > I run into the first issue here when the index has not completed > > loading from the local gateway. This causes the count query fail. If I > > wait and retry the count query it eventually completes with the > > expected answer and I can decide if I need to reindex the content. > > > > In the case the index does not exist I try to create a new index by > > executing a "create index" call: > > > > client.admin().indices().prepareCreate(INDEX\_NAME).execute(); > > > > Then immediately I want to customize a mapping but sometimes a "put > > mapping" call fails because the index is not fully created. Again if I > > wait long enough after the "create index" call the "put mapping" will > > succeed however it seems like there should be a better way. Any > > suggestions? > > > > The data set I am using is around 200-500 small documents, with 3-4 > > terms. I don't anticipate needing a second ES server any time soon so > > ideally any solution would not involve a second ES node. I have tried > > using the "local" and "file" gateway with similar results. Since the > > current dataset is small if I had to index each time the application > > started (no persistence of the index) that would be OK however I still > > have the "create index" / "put mapping" race condition. > > > > My current ES configuration is very simple: > > > > gateway.type=local > > gateway.recover\_after\_nodes=0 > > gateway.recover\_after\_time=0s > > > > Thanks in advance, > > > > Ryan ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [November 9, 2010, 1:25am UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/4 "2010-11-09T01:25:14Z")
Hi, you can have predefined custom types. Check [http://www.elasticsearch.com/docs/elasticsearch/mapping/builtin\_mappings/](http://www.elasticsearch.com/docs/elasticsearch/mapping/builtin_mappings/) [http://www.elasticsearch.com/docs/elasticsearch/mapping/builtin\_mappings/](http://www.elasticsearch.com/docs/elasticsearch/mapping/builtin_mappings/) Lukas On Tue, Nov 9, 2010 at 1:36 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > Thanks I will take a look. Is there no better way to structure this than > polling between API calls? How are people managing their custom type > mappings? > > Ryan > > On Mon, Nov 8, 2010 at 6:30 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > > > Hi, > > > > there is a cluster health API that you can use to check if new index is > > already allocated and ready for use. > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) > > > > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) > > Regards, > > Lukas > > > > On Tue, Nov 9, 2010 at 12:48 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > > > > > All, > > > > > > I am integrating Elastic Search into a new java/spring/hibernate/ > > > wicket application. I want to keep this application as standalone as > > > possible so I opted to integrate ES into the web application itself > > > (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has > > > been working great with the exception of "bootstrapping" the index at > > > startup. The current bootstrap flow looks like this: > > > > > > 1. Web application is started. > > > 2. Search service tries to run a count() query to determine if a full > > > index is needed. > > > > > > I run into the first issue here when the index has not completed > > > loading from the local gateway. This causes the count query fail. If I > > > wait and retry the count query it eventually completes with the > > > expected answer and I can decide if I need to reindex the content. > > > > > > In the case the index does not exist I try to create a new index by > > > executing a "create index" call: > > > > > > client.admin().indices().prepareCreate(INDEX\_NAME).execute(); > > > > > > Then immediately I want to customize a mapping but sometimes a "put > > > mapping" call fails because the index is not fully created. Again if I > > > wait long enough after the "create index" call the "put mapping" will > > > succeed however it seems like there should be a better way. Any > > > suggestions? > > > > > > The data set I am using is around 200-500 small documents, with 3-4 > > > terms. I don't anticipate needing a second ES server any time soon so > > > ideally any solution would not involve a second ES node. I have tried > > > using the "local" and "file" gateway with similar results. Since the > > > current dataset is small if I had to index each time the application > > > started (no persistence of the index) that would be OK however I still > > > have the "create index" / "put mapping" race condition. > > > > > > My current ES configuration is very simple: > > > > > > gateway.type=local > > > gateway.recover\_after\_nodes=0 > > > gateway.recover\_after\_time=0s > > > > > > Thanks in advance, > > > > > > Ryan ---
**Author:** ![jminard](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jminard/32/1848_2.png) [@jminard](https://discuss.elastic.co/u/jminard)\ **Post date:** [November 9, 2010, 12:37pm UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/5 "2010-11-09T12:37:07Z")
There have been conflicting suggestions around pushing mappings through the API vs. a configuration on the physical server. I think on a large cluster, ensuring file updates across the board is harder than pushing the mappings through the API. Especially if you create new index names dynamically but have index-specific mappings for them. Maybe an option to create and index synchronously? Option on apply mappings to be synchronous as well? Or back to polling to determine if it looks up to date... --j On 2010-11-08, at 11:25 PM, Lukáš Vlček wrote: > Hi, > > you can have predefined custom types. Check [http://www.elasticsearch.com/docs/elasticsearch/mapping/builtin\_mappings/](http://www.elasticsearch.com/docs/elasticsearch/mapping/builtin_mappings/) > > Lukas > > On Tue, Nov 9, 2010 at 1:36 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > Thanks I will take a look. Is there no better way to structure this than polling between API calls? How are people managing their custom type mappings? > > Ryan > > On Mon, Nov 8, 2010 at 6:30 PM, Lukáš Vlček [lukas.vlcek@gmail.com](mailto:lukas.vlcek@gmail.com) wrote: > Hi, > > there is a cluster health API that you can use to check if new index is already allocated and ready for use. > [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/cluster/health/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/cluster/health/) > > Regards, > Lukas > > On Tue, Nov 9, 2010 at 12:48 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > All, > > I am integrating Elastic Search into a new java/spring/hibernate/ > wicket application. I want to keep this application as standalone as > possible so I opted to integrate ES into the web application itself > (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has > been working great with the exception of "bootstrapping" the index at > startup. The current bootstrap flow looks like this: > > 1. Web application is started. > 2. Search service tries to run a count() query to determine if a full > index is needed. > > I run into the first issue here when the index has not completed > loading from the local gateway. This causes the count query fail. If I > wait and retry the count query it eventually completes with the > expected answer and I can decide if I need to reindex the content. > > In the case the index does not exist I try to create a new index by > executing a "create index" call: > > client.admin().indices().prepareCreate(INDEX\_NAME).execute(); > > Then immediately I want to customize a mapping but sometimes a "put > mapping" call fails because the index is not fully created. Again if I > wait long enough after the "create index" call the "put mapping" will > succeed however it seems like there should be a better way. Any > suggestions? > > The data set I am using is around 200-500 small documents, with 3-4 > terms. I don't anticipate needing a second ES server any time soon so > ideally any solution would not involve a second ES node. I have tried > using the "local" and "file" gateway with similar results. Since the > current dataset is small if I had to index each time the application > started (no persistence of the index) that would be OK however I still > have the "create index" / "put mapping" race condition. > > My current ES configuration is very simple: > > gateway.type=local > gateway.recover\_after\_nodes=0 > gateway.recover\_after\_time=0s > > Thanks in advance, > > Ryan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 9, 2010, 1:15pm UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/6 "2010-11-09T13:15:35Z")
Make sure to call get or actionGet once you execute an operation, otherwise, your not blocking on it to return (its executed in an async manner). Based on the code you posted, you don't call it on index creation. Regarding know when the index has recovered, you can use the cluster health API, as was suggested by Lukas. -shay.banon On Tue, Nov 9, 2010 at 1:48 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > All, > > I am integrating Elastic Search into a new java/spring/hibernate/ > wicket application. I want to keep this application as standalone as > possible so I opted to integrate ES into the web application itself > (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has > been working great with the exception of "bootstrapping" the index at > startup. The current bootstrap flow looks like this: > > 1. Web application is started. > 2. Search service tries to run a count() query to determine if a full > index is needed. > > I run into the first issue here when the index has not completed > loading from the local gateway. This causes the count query fail. If I > wait and retry the count query it eventually completes with the > expected answer and I can decide if I need to reindex the content. > > In the case the index does not exist I try to create a new index by > executing a "create index" call: > > client.admin().indices().prepareCreate(INDEX\_NAME).execute(); > > Then immediately I want to customize a mapping but sometimes a "put > mapping" call fails because the index is not fully created. Again if I > wait long enough after the "create index" call the "put mapping" will > succeed however it seems like there should be a better way. Any > suggestions? > > The data set I am using is around 200-500 small documents, with 3-4 > terms. I don't anticipate needing a second ES server any time soon so > ideally any solution would not involve a second ES node. I have tried > using the "local" and "file" gateway with similar results. Since the > current dataset is small if I had to index each time the application > started (no persistence of the index) that would be OK however I still > have the "create index" / "put mapping" race condition. > > My current ES configuration is very simple: > > gateway.type=local > gateway.recover\_after\_nodes=0 > gateway.recover\_after\_time=0s > > Thanks in advance, > > Ryan ---
**Author:** ![Ryan\_Crumley](https://avatars.discourse-cdn.com/v4/letter/r/b9e5f3/32.png) [@Ryan\_Crumley](https://discuss.elastic.co/u/Ryan_Crumley)\ **Post date:** [November 10, 2010, 3:44am UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/7 "2010-11-10T03:44:44Z")
Thanks for the information this got me on the right track. I ended up making two changes: - Before making a count query I am using the health api to wait until the index comes up: client.admin().cluster().prepareHealth(INDEX\_NAME).setWaitForYellowStatus().execute().actionGet(); If an IndexMissingException is thrown I know the index needs to be created. - When creating the index I had a problem with "put mapping" before the index was ready... Instead of using the health api you can also specify a mapping when the index is created. Thanks again, Ryan On Tue, Nov 9, 2010 at 7:15 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Make sure to call get or actionGet once you execute an operation, > otherwise, your not blocking on it to return (its executed in an async > manner). Based on the code you posted, you don't call it on index creation. > > Regarding know when the index has recovered, you can use the cluster health > API, as was suggested by Lukas. > > -shay.banon > > On Tue, Nov 9, 2010 at 1:48 AM, Ryan Crumley [crumley@gmail.com](mailto:crumley@gmail.com) wrote: > > > All, > > > > I am integrating Elastic Search into a new java/spring/hibernate/ > > wicket application. I want to keep this application as standalone as > > possible so I opted to integrate ES into the web application itself > > (using NodeBuilder.nodeBuilder().data(true).local(false) ). This has > > been working great with the exception of "bootstrapping" the index at > > startup. The current bootstrap flow looks like this: > > > > 1. Web application is started. > > 2. Search service tries to run a count() query to determine if a full > > index is needed. > > > > I run into the first issue here when the index has not completed > > loading from the local gateway. This causes the count query fail. If I > > wait and retry the count query it eventually completes with the > > expected answer and I can decide if I need to reindex the content. > > > > In the case the index does not exist I try to create a new index by > > executing a "create index" call: > > > > client.admin().indices().prepareCreate(INDEX\_NAME).execute(); > > > > Then immediately I want to customize a mapping but sometimes a "put > > mapping" call fails because the index is not fully created. Again if I > > wait long enough after the "create index" call the "put mapping" will > > succeed however it seems like there should be a better way. Any > > suggestions? > > > > The data set I am using is around 200-500 small documents, with 3-4 > > terms. I don't anticipate needing a second ES server any time soon so > > ideally any solution would not involve a second ES node. I have tried > > using the "local" and "file" gateway with similar results. Since the > > current dataset is small if I had to index each time the application > > started (no persistence of the index) that would be OK however I still > > have the "create index" / "put mapping" race condition. > > > > My current ES configuration is very simple: > > > > gateway.type=local > > gateway.recover\_after\_nodes=0 > > gateway.recover\_after\_time=0s > > > > Thanks in advance, > > > > Ryan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:16am UTC](https://discuss.elastic.co/t/index-bootstrapping/3537/8 "2017-07-06T04:16:50Z")