# Race condition when retrieving the mapping immediately after creating a document that contains a new field **URL:** \ **Category:** Elasticsearch\ **Created:** [March 2, 2015, 9:44pm UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481 "2015-03-02T21:44:38Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Andrew\_Pak](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Andrew\_Pak](https://discuss.elastic.co/u/Andrew_Pak)\ **Post date:** [March 2, 2015, 9:44pm UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481/1 "2015-03-02T21:44:38Z")
Hello! Here's my issue: 1. I create an index with a mapping that has a dynamic template. 2. I create a new document that contains a new field that will match a dynamic template in the mapping and that will be added to the mapping. The PUT succeeds with 201 Created. 3. I then retrieve the mapping with a GET to check that the new field has been correctly added. The GET succeeds with 200 OK. 4. The returned mapping doesn't always contain the newly created field. It sometimes takes a small delay (\< 1s) for the updated mapping to be returned. Seems like a race condition, so I'm wondering if this is expected behaviour and if there is a setting or parameter that can be set to guarantee that the mapping gets refreshed atomically, just like you can force-refresh an index. This is not currently a show-stopping issue, I'd just like to have a definitive answer. The reason I need the mapping to reflect the changes immediately is that I have unit tests that depend on field information retrieved from the mapping at run-time and they sporadically fail because the returned mapping and the expected mapping don't always match up. Here's a curl recreation script: > However, it seems that curl is slow enough that the small delay needed for the mapping to refresh passes in-between curl calls and the script doesn't actually reproduce the issue. I hence created an equivalent Python script that retries the whole process until the issue is demonstrated: > Here's a sample output from the script (contains a few iterations of the entire recreation cycle): > /bin/python es\_mapping\_race\_condition.py > Delete index /documents: {"acknowledged":true} > Create index /documents: {"acknowledged":true} > Create mapping /documents/pdf: {"acknowledged":true} > Create new /documents/pdf document: > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwZgoOHS3xjDkoFH","\_version":1,"created":true} > Refresh index /documents {"\_shards":{"total":10,"successful":5,"failed":0}} > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', > u'title\_str', u'title\_str.tags\_str'] > Delete index /documents: {"acknowledged":true} > Create index /documents: {"acknowledged":true} > Create mapping /documents/pdf: {"acknowledged":true} > Create new /documents/pdf document: > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwsCoOHS3xjDkoFI","\_version":1,"created":true} > Refresh index /documents {"\_shards":{"total":10,"successful":5,"failed":0}} > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', > u'title\_str', u'title\_str.tags\_str'] > Delete index /documents: {"acknowledged":true} > Create index /documents: {"acknowledged":true} > Create mapping /documents/pdf: {"acknowledged":true} > Create new /documents/pdf document: > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWxDioOHS3xjDkoFJ","\_version":1,"created":true} > Refresh index /documents {"\_shards":{"total":10,"successful":5,"failed":0}} > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > ######################################################################## # # Field 'new\_field\_str' was not found in the mapping. # ######################################################################## Thanks in advance for any help! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 2, 2015, 10:03pm UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481/2 "2015-03-02T22:03:46Z")
You have to wait for green health of the index before retrieving the new mapping. > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. While the mapping changes, cluster has some work to do to propagate the new field to all nodes. This takes some time. It is not a race condition. Jörg On Mon, Mar 2, 2015 at 10:44 PM, Andrew Pak [andrew.pak.work@gmail.com](mailto:andrew.pak.work@gmail.com) wrote: > Hello! > > Here's my issue: > > 1. I create an index with a mapping that has a dynamic template. > 2. I create a new document that contains a new field that will match a > dynamic template in the mapping and that will be added to the mapping. The > PUT succeeds with 201 Created. > 3. I then retrieve the mapping with a GET to check that the new field has > been correctly added. The GET succeeds with 200 OK. > 4. The returned mapping doesn't always contain the newly created field. It > sometimes takes a small delay (\< 1s) for the updated mapping to be returned. > > Seems like a race condition, so I'm wondering if this is expected > behaviour and if there is a setting or parameter that can be set to > guarantee that the mapping gets refreshed atomically, just like you can > force-refresh an index. This is not currently a show-stopping issue, I'd > just like to have a definitive answer. > > The reason I need the mapping to reflect the changes immediately is that I > have unit tests that depend on field information retrieved from the mapping > at run-time and they sporadically fail because the returned mapping and the > expected mapping don't always match up. > > Here's a curl recreation script: > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/7f116867603a44575ce1) > > However, it seems that curl is slow enough that the small delay needed for > the mapping to refresh passes in-between curl calls and the script doesn't > actually reproduce the issue. I hence created an equivalent Python script > that retries the whole process until the issue is demonstrated: > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb) > > Here's a sample output from the script (contains a few iterations of the > entire recreation cycle): > > > /bin/python es\_mapping\_race\_condition.py > > Delete index /documents: {"acknowledged":true} > > Create index /documents: {"acknowledged":true} > > Create mapping /documents/pdf: {"acknowledged":true} > > Create new /documents/pdf document: > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwZgoOHS3xjDkoFH","\_version":1,"created":true} > > Refresh index /documents {"\_shards":{"total":10,"successful":5,"failed":0}} > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', u'title\_str', u'title\_str.tags\_str'] > > Delete index /documents: {"acknowledged":true} > > Create index /documents: {"acknowledged":true} > > Create mapping /documents/pdf: {"acknowledged":true} > > Create new /documents/pdf document: > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwsCoOHS3xjDkoFI","\_version":1,"created":true} > > Refresh index /documents {"\_shards":{"total":10,"successful":5,"failed":0}} > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', u'title\_str', u'title\_str.tags\_str'] > > Delete index /documents: {"acknowledged":true} > > Create index /documents: {"acknowledged":true} > > Create mapping /documents/pdf: {"acknowledged":true} > > Create new /documents/pdf document: > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWxDioOHS3xjDkoFJ","\_version":1,"created":true} > > Refresh index /documents {"\_shards":{"total":10,"successful":5,"failed":0}} > > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > > ######################################################################## > > # > > # Field 'new\_field\_str' was not found in the mapping. > > # > > ######################################################################## > > Thanks in advance for any help! > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFy%2B7gLoV6yF2VutUWwDveSxC2ccVK2%2BBQdBRqkD2cAVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFy%2B7gLoV6yF2VutUWwDveSxC2ccVK2%2BBQdBRqkD2cAVg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andrew\_Pak](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Andrew\_Pak](https://discuss.elastic.co/u/Andrew_Pak)\ **Post date:** [March 3, 2015, 3:34pm UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481/3 "2015-03-03T15:34:49Z")
Thanks for your reply Jörg! However, even after I wait for green status, I still get the issue, here's the output of the updated script that does a synchronous call to get the health of the cluster with the 'wait\_for\_status=green' parameter. /bin/python es\_dynamic\_mapping\_test.py Delete index /documents: {"acknowledged":true} Create index /documents: {"acknowledged":true} Create mapping /documents/pdf: {"acknowledged":true} Create new /documents/pdf document: {"\_index":"documents","\_type":"pdf","\_id":"AUvgNoIq70doA8\_gZ7fd","\_version":1,"created":true} Refresh index /documents {"\_shards":{"total":1,"successful":1,"failed":0}} Wait for green status: {"cluster\_name":"helixdev","status":"green","timed\_out":false,"number\_of\_nodes":2,"number\_of\_data\_nodes":2,"active\_primary\_shar ds":1,"active\_shards":1,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0} Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] ######################################################################## # # Field 'new\_field\_str' was not found in the mapping. # ######################################################################## > Thanks again! Andrew On Monday, March 2, 2015 at 5:04:05 PM UTC-5, Jörg Prante wrote: > You have to wait for green health of the index before retrieving the new > mapping. > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html) > > While the mapping changes, cluster has some work to do to propagate the > new field to all nodes. This takes some time. It is not a race condition. > > Jörg > > On Mon, Mar 2, 2015 at 10:44 PM, Andrew Pak \<[andrew....@gmail.com](mailto:andrew....@gmail.com) > \\> wrote: > > > Hello! > > > > Here's my issue: > > > > 1. I create an index with a mapping that has a dynamic template. > > 2. I create a new document that contains a new field that will match a > > dynamic template in the mapping and that will be added to the mapping. The > > PUT succeeds with 201 Created. > > 3. I then retrieve the mapping with a GET to check that the new field has > > been correctly added. The GET succeeds with 200 OK. > > 4. The returned mapping doesn't always contain the newly created field. > > It sometimes takes a small delay (\< 1s) for the updated mapping to be > > returned. > > > > Seems like a race condition, so I'm wondering if this is expected > > behaviour and if there is a setting or parameter that can be set to > > guarantee that the mapping gets refreshed atomically, just like you can > > force-refresh an index. This is not currently a show-stopping issue, I'd > > just like to have a definitive answer. > > > > The reason I need the mapping to reflect the changes immediately is that > > I have unit tests that depend on field information retrieved from the > > mapping at run-time and they sporadically fail because the returned mapping > > and the expected mapping don't always match up. > > > > Here's a curl recreation script: > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/7f116867603a44575ce1) > > > > However, it seems that curl is slow enough that the small delay needed > > for the mapping to refresh passes in-between curl calls and the script > > doesn't actually reproduce the issue. I hence created an equivalent Python > > script that retries the whole process until the issue is demonstrated: > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb) > > > > Here's a sample output from the script (contains a few iterations of the > > entire recreation cycle): > > > > > /bin/python es\_mapping\_race\_condition.py > > > Delete index /documents: {"acknowledged":true} > > > Create index /documents: {"acknowledged":true} > > > Create mapping /documents/pdf: {"acknowledged":true} > > > Create new /documents/pdf document: > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwZgoOHS3xjDkoFH","\_version":1,"created":true} > > > Refresh index /documents > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', > > > u'title\_str', u'title\_str.tags\_str'] > > > Delete index /documents: {"acknowledged":true} > > > Create index /documents: {"acknowledged":true} > > > Create mapping /documents/pdf: {"acknowledged":true} > > > Create new /documents/pdf document: > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwsCoOHS3xjDkoFI","\_version":1,"created":true} > > > Refresh index /documents > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', > > > u'title\_str', u'title\_str.tags\_str'] > > > Delete index /documents: {"acknowledged":true} > > > Create index /documents: {"acknowledged":true} > > > Create mapping /documents/pdf: {"acknowledged":true} > > > Create new /documents/pdf document: > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWxDioOHS3xjDkoFJ","\_version":1,"created":true} > > > Refresh index /documents > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > > > ######################################################################## > > > > # > > > > # Field 'new\_field\_str' was not found in the mapping. > > > > # > > > > ######################################################################## > > > > Thanks in advance for any help! > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/84bcedf1-c83a-47b7-a28b-a97258c4ff3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/84bcedf1-c83a-47b7-a28b-a97258c4ff3f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andrew\_Pak](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Andrew\_Pak](https://discuss.elastic.co/u/Andrew_Pak)\ **Post date:** [March 3, 2015, 4:44pm UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481/4 "2015-03-03T16:44:33Z")
More info: I've tested it with ES versions 1.4.2, 1.4.3, and 1.4.4, and the reproduction script demonstrates the issue on all of them. Sometimes it happens right away, other times you have to wait for a few minutes/iterations, but each time the mapping is retrieved is after the status of the cluster is green. Here's my test environment: Windows 7 Enterprise, default configuration ES with two nodes. Java version: jre1.8.0\_31. On Tuesday, March 3, 2015 at 10:34:49 AM UTC-5, Andrew Pak wrote: > Thanks for your reply Jörg! However, even after I wait for green status, I > still get the issue, here's the output of the updated script that does a > synchronous call to get the health of the cluster with the > 'wait\_for\_status=green' parameter. > > /bin/python es\_dynamic\_mapping\_test.py > Delete index /documents: {"acknowledged":true} > Create index /documents: {"acknowledged":true} > Create mapping /documents/pdf: {"acknowledged":true} > Create new /documents/pdf document: > {"\_index":"documents","\_type":"pdf","\_id":"AUvgNoIq70doA8\_gZ7fd","\_version":1,"created":true} > Refresh index /documents {"\_shards":{"total":1,"successful":1,"failed":0}} > Wait for green status: > {"cluster\_name":"helixdev","status":"green","timed\_out":false,"number\_of\_nodes":2,"number\_of\_data\_nodes":2,"active\_primary\_shar > > ds":1,"active\_shards":1,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0} > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > ######################################################################## > > # > > # Field 'new\_field\_str' was not found in the mapping. > > # > > ######################################################################## > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb) > > Thanks again! > Andrew > > On Monday, March 2, 2015 at 5:04:05 PM UTC-5, Jörg Prante wrote: > > > You have to wait for green health of the index before retrieving the new > > mapping. > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html) > > > > While the mapping changes, cluster has some work to do to propagate the > > new field to all nodes. This takes some time. It is not a race condition. > > > > Jörg > > > > On Mon, Mar 2, 2015 at 10:44 PM, Andrew Pak [andrew....@gmail.com](mailto:andrew....@gmail.com) wrote: > > > > > Hello! > > > > > > Here's my issue: > > > > > > 1. I create an index with a mapping that has a dynamic template. > > > 2. I create a new document that contains a new field that will match a > > > dynamic template in the mapping and that will be added to the mapping. The > > > PUT succeeds with 201 Created. > > > 3. I then retrieve the mapping with a GET to check that the new field > > > has been correctly added. The GET succeeds with 200 OK. > > > 4. The returned mapping doesn't always contain the newly created field. > > > It sometimes takes a small delay (\< 1s) for the updated mapping to be > > > returned. > > > > > > Seems like a race condition, so I'm wondering if this is expected > > > behaviour and if there is a setting or parameter that can be set to > > > guarantee that the mapping gets refreshed atomically, just like you can > > > force-refresh an index. This is not currently a show-stopping issue, I'd > > > just like to have a definitive answer. > > > > > > The reason I need the mapping to reflect the changes immediately is that > > > I have unit tests that depend on field information retrieved from the > > > mapping at run-time and they sporadically fail because the returned mapping > > > and the expected mapping don't always match up. > > > > > > Here's a curl recreation script: > > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/7f116867603a44575ce1) > > > > > > However, it seems that curl is slow enough that the small delay needed > > > for the mapping to refresh passes in-between curl calls and the script > > > doesn't actually reproduce the issue. I hence created an equivalent Python > > > script that retries the whole process until the issue is demonstrated: > > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb) > > > > > > Here's a sample output from the script (contains a few iterations of the > > > entire recreation cycle): > > > > > > > /bin/python es\_mapping\_race\_condition.py > > > > Delete index /documents: {"acknowledged":true} > > > > Create index /documents: {"acknowledged":true} > > > > Create mapping /documents/pdf: {"acknowledged":true} > > > > Create new /documents/pdf document: > > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwZgoOHS3xjDkoFH","\_version":1,"created":true} > > > > Refresh index /documents > > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', u'title\_str', u'title\_str.tags\_str'] > > > > Delete index /documents: {"acknowledged":true} > > > > Create index /documents: {"acknowledged":true} > > > > Create mapping /documents/pdf: {"acknowledged":true} > > > > Create new /documents/pdf document: > > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwsCoOHS3xjDkoFI","\_version":1,"created":true} > > > > Refresh index /documents > > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', u'title\_str', u'title\_str.tags\_str'] > > > > Delete index /documents: {"acknowledged":true} > > > > Create index /documents: {"acknowledged":true} > > > > Create mapping /documents/pdf: {"acknowledged":true} > > > > Create new /documents/pdf document: > > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWxDioOHS3xjDkoFJ","\_version":1,"created":true} > > > > Refresh index /documents > > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > > > > ######################################################################## > > > > > > # > > > > > > # Field 'new\_field\_str' was not found in the mapping. > > > > > > # > > > > > > ######################################################################## > > > > > > Thanks in advance for any help! > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/50af4889-5bfa-41e2-8c1a-acffebe7d6fb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/50af4889-5bfa-41e2-8c1a-acffebe7d6fb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andrew\_Pak](https://avatars.discourse-cdn.com/v4/letter/a/ee59a6/32.png) [@Andrew\_Pak](https://discuss.elastic.co/u/Andrew_Pak)\ **Post date:** [March 4, 2015, 10:22pm UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481/5 "2015-03-04T22:22:06Z")
Just to add a TL;DR: I'd like to know definitively whether there is anything I can do to force a mapping refresh and to guarantee that following an addition of a new field, retrieving the mapping will always produce a mapping that contains that field. Thanks in advance for any insight! On Tuesday, March 3, 2015 at 11:44:33 AM UTC-5, Andrew Pak wrote: > More info: I've tested it with ES versions 1.4.2, 1.4.3, and 1.4.4, and > the reproduction script demonstrates the issue on all of them. Sometimes it > happens right away, other times you have to wait for a few > minutes/iterations, but each time the mapping is retrieved is after the > status of the cluster is green. > > Here's my test environment: Windows 7 Enterprise, default configuration ES > with two nodes. Java version: jre1.8.0\_31. > > On Tuesday, March 3, 2015 at 10:34:49 AM UTC-5, Andrew Pak wrote: > > > Thanks for your reply Jörg! However, even after I wait for green status, > > I still get the issue, here's the output of the updated script that does a > > synchronous call to get the health of the cluster with the > > 'wait\_for\_status=green' parameter. > > > > /bin/python es\_dynamic\_mapping\_test.py > > Delete index /documents: {"acknowledged":true} > > Create index /documents: {"acknowledged":true} > > Create mapping /documents/pdf: {"acknowledged":true} > > Create new /documents/pdf document: > > {"\_index":"documents","\_type":"pdf","\_id":"AUvgNoIq70doA8\_gZ7fd","\_version":1,"created":true} > > Refresh index /documents {"\_shards":{"total":1,"successful":1,"failed":0}} > > Wait for green status: > > {"cluster\_name":"helixdev","status":"green","timed\_out":false,"number\_of\_nodes":2,"number\_of\_data\_nodes":2,"active\_primary\_shar > > > > ds":1,"active\_shards":1,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0} > > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > > ######################################################################## > > > > # > > > > # Field 'new\_field\_str' was not found in the mapping. > > > > # > > > > ######################################################################## > > > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb) > > > > Thanks again! > > Andrew > > > > On Monday, March 2, 2015 at 5:04:05 PM UTC-5, Jörg Prante wrote: > > > > > You have to wait for green health of the index before retrieving the new > > > mapping. > > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-health.html) > > > > > > While the mapping changes, cluster has some work to do to propagate the > > > new field to all nodes. This takes some time. It is not a race condition. > > > > > > Jörg > > > > > > On Mon, Mar 2, 2015 at 10:44 PM, Andrew Pak [andrew....@gmail.com](mailto:andrew....@gmail.com) > > > wrote: > > > > > > > Hello! > > > > > > > > Here's my issue: > > > > > > > > 1. I create an index with a mapping that has a dynamic template. > > > > 2. I create a new document that contains a new field that will match a > > > > dynamic template in the mapping and that will be added to the mapping. The > > > > PUT succeeds with 201 Created. > > > > 3. I then retrieve the mapping with a GET to check that the new field > > > > has been correctly added. The GET succeeds with 200 OK. > > > > 4. The returned mapping doesn't always contain the newly created field. > > > > It sometimes takes a small delay (\< 1s) for the updated mapping to be > > > > returned. > > > > > > > > Seems like a race condition, so I'm wondering if this is expected > > > > behaviour and if there is a setting or parameter that can be set to > > > > guarantee that the mapping gets refreshed atomically, just like you can > > > > force-refresh an index. This is not currently a show-stopping issue, I'd > > > > just like to have a definitive answer. > > > > > > > > The reason I need the mapping to reflect the changes immediately is > > > > that I have unit tests that depend on field information retrieved from the > > > > mapping at run-time and they sporadically fail because the returned mapping > > > > and the expected mapping don't always match up. > > > > > > > > Here's a curl recreation script: > > > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/7f116867603a44575ce1) > > > > > > > > However, it seems that curl is slow enough that the small delay needed > > > > for the mapping to refresh passes in-between curl calls and the script > > > > doesn't actually reproduce the issue. I hence created an equivalent Python > > > > script that retries the whole process until the issue is demonstrated: > > > > [Script to reproduce the issue where a adding a new field to a dynamic mapping does not atomically refresh the mapping · GitHub](https://gist.github.com/elasticsearcher/2dff7ee44e11c6a39bfb) > > > > > > > > Here's a sample output from the script (contains a few iterations of > > > > the entire recreation cycle): > > > > > > > > > /bin/python es\_mapping\_race\_condition.py > > > > > Delete index /documents: {"acknowledged":true} > > > > > Create index /documents: {"acknowledged":true} > > > > > Create mapping /documents/pdf: {"acknowledged":true} > > > > > Create new /documents/pdf document: > > > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwZgoOHS3xjDkoFH","\_version":1,"created":true} > > > > > Refresh index /documents > > > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', u'title\_str', u'title\_str.tags\_str'] > > > > > Delete index /documents: {"acknowledged":true} > > > > > Create index /documents: {"acknowledged":true} > > > > > Create mapping /documents/pdf: {"acknowledged":true} > > > > > Create new /documents/pdf document: > > > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWwsCoOHS3xjDkoFI","\_version":1,"created":true} > > > > > Refresh index /documents > > > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > Retrieved fields: [u'new\_field\_str', u'new\_field\_str.tags\_str', u'title\_str', u'title\_str.tags\_str'] > > > > > Delete index /documents: {"acknowledged":true} > > > > > Create index /documents: {"acknowledged":true} > > > > > Create mapping /documents/pdf: {"acknowledged":true} > > > > > Create new /documents/pdf document: > > > > > {"\_index":"documents","\_type":"pdf","\_id":"AUvcWxDioOHS3xjDkoFJ","\_version":1,"created":true} > > > > > Refresh index /documents > > > > > {"\_shards":{"total":10,"successful":5,"failed":0}} > > > > > Retrieved fields: [u'title\_str', u'title\_str.tags\_str'] > > > > > ######################################################################## > > > > > > > > # > > > > > > > > # Field 'new\_field\_str' was not found in the mapping. > > > > > > > > # > > > > > > > > ######################################################################## > > > > > > > > Thanks in advance for any help! > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/88e7df12-e862-442b-8e15-3c2a6fcae7d4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/59e9e835-7706-4e09-8a21-fdaf3df4369c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/59e9e835-7706-4e09-8a21-fdaf3df4369c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:28am UTC](https://discuss.elastic.co/t/race-condition-when-retrieving-the-mapping-immediately-after-creating-a-document-that-contains-a-new-field/22481/6 "2017-07-06T00:28:29Z")