# Failure during refresh-mapping 0.90.5 **URL:** \ **Category:** Elasticsearch\ **Created:** [November 6, 2013, 4:10pm UTC](https://discuss.elastic.co/t/failure-during-refresh-mapping-0-90-5/14291 "2013-11-06T16:10:32Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![pecke01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pecke01/32/1995_2.png) [@pecke01](https://discuss.elastic.co/u/pecke01)\ **Post date:** [November 6, 2013, 4:10pm UTC](https://discuss.elastic.co/t/failure-during-refresh-mapping-0-90-5/14291/1 "2013-11-06T16:10:32Z")
Hi, After an upgrade to 90.5 we are seeing some issues that we find concerning in the logs. [2013-11-06 13:49:54,587][WARN][cluster.metadata] [NODENAME] failure during [refresh-mapping [INDEXNAME][[TYPENAMES]]] java.lang.NullPointerException I have browsed the code but can't at least in master see where this issue might occur exactly. I am wondering what the issue might be and if someone else have seen this. Best Regards Marcus Granström -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [November 6, 2013, 4:59pm UTC](https://discuss.elastic.co/t/failure-during-refresh-mapping-0-90-5/14291/2 "2013-11-06T16:59:47Z")
hey can you try to figure out where this is coming from or maybe provide a recreation? simon On Wednesday, November 6, 2013 5:10:32 PM UTC+1, Marcus Granstrom wrote: > Hi, > > After an upgrade to 90.5 we are seeing some issues that we find concerning > in the logs. > > [2013-11-06 13:49:54,587][WARN][cluster.metadata] [NODENAME] failure > during [refresh-mapping [INDEXNAME][[TYPENAMES]]] > java.lang.NullPointerException > > I have browsed the code but can't at least in master see where this issue > might occur exactly. I am wondering what the issue might be and if someone > else have seen this. > > Best Regards > > Marcus Granström -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Henrik\_Lindstrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henrik_lindstrom/32/2001_2.png) [@Henrik\_Lindstrom](https://discuss.elastic.co/u/Henrik_Lindstrom)\ **Post date:** [November 7, 2013, 7:41am UTC](https://discuss.elastic.co/t/failure-during-refresh-mapping-0-90-5/14291/3 "2013-11-07T07:41:10Z")
It might be related to: > > > For a maintenance operation we disabled allocation and restart one node. > > \`\`\` > $ …curl -XPUT localhost:9023/\_cluster/settings -d '{"transient":{"cluster.routing.allocation.disable\_allocation":"true"}} > \# stop the node tpsmdt03s > \# start the node tpsmdt03s > $ curl -XPUT localhost:9023/\_cluster/settings -d '{"transient": "cluster.routing.allocation.disable\_allocation":"false"}} > \`\`\` > > Unfortunately, something went wrong with ES: > > \`\`\` > \[2012-11-28 11:02:02,324\]\[INFO \]\[cluster.routing.allocation.decider\] \[tpsmdt01s\] updating \[cluster.routing.allocation.disable\_allocation\] from \[false\] to \[true\] > \[2012-11-28 11:08:13,170\]\[INFO \]\[discovery.zen \] \[tpsmdt01s\] master\_left \[\[tpsmdt03s\]\[6Sd72NOiTOayO8nPtlH9dg\]\[inet\[/10.26.165.12:9027\]\]\], reason \[transport disconnected (with verified connect)\] > \[2012-11-28 11:08:13,177\]\[INFO \]\[cluster.service \] \[tpsmdt01s\] master {new \[tpsmdt01s\]\[JIjtYHpkQBa8\_GnvcLM4Nw\]\[inet\[/10.26.165.10:9027\]\], previous \[tpsmdt03s\]\[6Sd72NOiTOayO8nPtlH9dg\]\[inet\[/10.26.165.12:9027\]\]}, removed {\[tpsmdt03s\]\[6Sd72NOiTOayO8nPtlH9dg\]\[inet\[/10.26.165.12:9027\]\],}, reason: zen-disco-master\_failed (\[tpsmdt03s\]\[6Sd72NOiTOayO8nPtlH9dg\]\[inet\[/10.26.165.12:9027\]\]) > ... > \[2012-11-28 11:10:02,449\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] \[4f072ea3a623374256000001\] update\_mapping \[50b5e2f946b88e97370000e5\] (dynamic) > \[2012-11-28 11:17:12,867\]\[INFO \]\[cluster.service \] \[tpsmdt01s\] added {\[tpsmdt03s\]\[m8ckMTZlQ1Kuvd8P7RJtfg\]\[inet\[/10.26.165.12:9027\]\],}, reason: zen-disco-receive(join from node\[\[tpsmdt03s\]\[m8ckMTZlQ1Kuvd8P7RJtfg\]\[inet\[/10.26.165.12:9027\]\]\]) > \[2012-11-28 11:18:23,379\]\[INFO \]\[cluster.routing.allocation.decider\] \[tpsmdt01s\] updating \[cluster.routing.allocation.disable\_allocation\] from \[true\] to \[false\] > ... > 2012-11-28 14:40:02,615\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] \[4f072ea3a623374256000001\] update\_mapping \[50b6143246b88e9737000172\] (dynamic) > \[2012-11-28 14:40:04,885\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] \[\[4f072ea3a623374256000001\]\] remove\_mapping \[50b6143246b88e9737000172\] > \[2012-11-28 14:45:02,056\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] > ... > \[4f072ea3a623374256000001\] update\_mapping \[50b6155d46b88e9737000185\] (dynamic) > \[2012-11-28 14:45:04,359\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] \[\[4f072ea3a623374256000001\]\] remove\_mapping \[50b6155d46b88e9737000185\] > \[2012-11-28 14:45:04,482\]\[WARN \]\[cluster.metadata \] \[tpsmdt01s\] failed to dynamically refresh the mapping in cluster\_state from shard > java.lang.NullPointerException > at org.elasticsearch.cluster.metadata.MetaDataMappingService$1.execute(MetaDataMappingService.java:130) > at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:208) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > \`\`\` > > And now, I have a lot of WARN > > \`\`\` > \[2013-02-08 10:45:04,324\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] \[\[4f072ea3a623374256000001\]\] remove\_mapping \[5114c91de5c0e0950f0000d2\] > \[2013-02-08 10:45:04,459\]\[INFO \]\[cluster.metadata \] \[tpsmdt01s\] \[\[4f072ea3a623374256000001\]\] remove\_mapping \[5114c91d511a3418550000d4\] > \[2013-02-08 10:45:06,599\]\[WARN \]\[cluster.metadata \] \[tpsmdt01s\] failed to dynamically refresh the mapping in cluster\_state from shard > java.lang.NullPointerException > \[2013-02-08 10:45:06,964\]\[WARN \]\[cluster.metadata \] \[tpsmdt01s\] failed to dynamically refresh the mapping in cluster\_state from shard > java.lang.NullPointerException > \`\`\` > > When I say a lot, I mean it ;) > > \`\`\` > $ grep 'failed to dynamically refresh the mapping in cluster\_state from shard' search-0.19\* | wc -l > 58051 > \`\`\` That also happens when refresing mappings (the exact error message has changed between 0.19. and 0.90) On Wednesday, 6 November 2013 17:59:47 UTC+1, simonw wrote: > hey can you try to figure out where this is coming from or maybe provide a > recreation? > > simon > > On Wednesday, November 6, 2013 5:10:32 PM UTC+1, Marcus Granstrom wrote: > > > Hi, > > > > After an upgrade to 90.5 we are seeing some issues that we find > > concerning in the logs. > > > > [2013-11-06 13:49:54,587][WARN][cluster.metadata] [NODENAME] failure > > during [refresh-mapping [INDEXNAME][[TYPENAMES]]] > > java.lang.NullPointerException > > > > I have browsed the code but can't at least in master see where this issue > > might occur exactly. I am wondering what the issue might be and if someone > > else have seen this. > > > > Best Regards > > > > Marcus Granström -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Henrik\_Lindstrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henrik_lindstrom/32/2001_2.png) [@Henrik\_Lindstrom](https://discuss.elastic.co/u/Henrik_Lindstrom)\ **Post date:** [November 7, 2013, 2:28pm UTC](https://discuss.elastic.co/t/failure-during-refresh-mapping-0-90-5/14291/4 "2013-11-07T14:28:57Z")
When processing the mappings I can see from the debug output that the parsed differs from the original. When looking in the source I then see: IndicesClusterStateService.java:412 // we don't apply default, since it has been applied when the mappings were parsed initially mapperService.merge(mappingType, mappingSource.string(), false); if (!mapperService.documentMapper(mappingType).mappingSource().equals(mappingSource)) { requiresRefresh = true; // this might happen when upgrading from 0.15 to 0.16 logger.debug("[{}] parsed mapping [{}], and got different sources\noriginal:\n{}\nparsed:\n{}", index, mappingType, mappingSource, mapperService.documentMapper(mappingType).mappingSource()); } Is that comment correct? (In my interpretation this should only happen when upgrading from 0.15 to 0.16) We actually see this happen in 0.90.5 when we have created the index in 0.90.5. Another strange thing is that even though the mappings are processed for each cluster change we don't get the error every time for a specific index and type. On Thursday, 7 November 2013 08:41:10 UTC+1, Henrik Lindström wrote: > It might be related to: > [NullPointerException failed to dynamically refresh the mapping in cluster\_state from shard · Issue #2633 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2633) > > That also happens when refresing mappings (the exact error message has > changed between 0.19. and 0.90) > > On Wednesday, 6 November 2013 17:59:47 UTC+1, simonw wrote: > > > hey can you try to figure out where this is coming from or maybe provide > > a recreation? > > > > simon > > > > On Wednesday, November 6, 2013 5:10:32 PM UTC+1, Marcus Granstrom wrote: > > > > > Hi, > > > > > > After an upgrade to 90.5 we are seeing some issues that we find > > > concerning in the logs. > > > > > > [2013-11-06 13:49:54,587][WARN][cluster.metadata] [NODENAME] failure > > > during [refresh-mapping [INDEXNAME][[TYPENAMES]]] > > > java.lang.NullPointerException > > > > > > I have browsed the code but can't at least in master see where this > > > issue might occur exactly. I am wondering what the issue might be and if > > > someone else have seen this. > > > > > > Best Regards > > > > > > Marcus Granström -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:08am UTC](https://discuss.elastic.co/t/failure-during-refresh-mapping-0-90-5/14291/5 "2017-07-06T02:08:20Z")