# Problems with 0.18.X **URL:** \ **Category:** Elasticsearch\ **Created:** [November 21, 2011, 10:07pm UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933 "2011-11-21T22:07:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Grant](https://avatars.discourse-cdn.com/v4/letter/g/a5b964/32.png) [@Grant](https://discuss.elastic.co/u/Grant)\ **Post date:** [November 21, 2011, 10:07pm UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933/1 "2011-11-21T22:07:06Z")
Today we attempted an upgrade from 0.17.6 to 0.18.4. We run a 4 node cluster with lots of small indexes (roughly 0-800mb each) each with 2 replicas, no sharding. We had about 3gb of data on each node at the time. After a full restart, the cluster failed to recover and remained in a red state until nodes would start to become unresponsive. This happened as soon as the processes resident size equaled its JVM heap size (2gb initially). Raising the heap to 8gb had no effect. ES would just consume all the available heap then become unresponsive after having done so. We eventually decided to re-index (starting with an empty data directory on all nodes), still on 0.18.4, and discovered that we could create about 10 (empty) indexes in rapid succession and then ES would become unresponsive. It was also very slow to move out of red or yellow states. Having lost most hope by this point, we downgraded to 0.17.6 and began indexing again, and everything flew along in typical fashion. Anyone have any thoughts as to what gives? The only useful log messages I turned up was when we were trying to re- index from scratch on 0.18.4... these streamed continuously: [2011-11-21 21:46:19,421][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,485][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,544][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,605][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,670][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,730][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,792][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,857][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,916][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:19,975][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,043][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,102][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,163][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,224][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,292][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,351][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,411][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,474][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,534][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] [2011-11-21 21:46:20,594][WARN][cluster.metadata] [prod- elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with cluster state for types [[contact\_document]] ---
**Author:** ![Matt1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matt1/32/2986_2.png) [@Matt1](https://discuss.elastic.co/u/Matt1)\ **Post date:** [November 22, 2011, 3:44am UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933/2 "2011-11-22T03:44:51Z")
I was having the same problem, which Shay replied: [https://groups.google.com/d/topic/elasticsearch/cYmNcpV6g98/discussion](https://groups.google.com/d/topic/elasticsearch/cYmNcpV6g98/discussion) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 22, 2011, 7:56am UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933/3 "2011-11-22T07:56:24Z")
Hey, You should not see the "re-sync" message when indexing into a fresh 0.18.4 cluster, are you sure you saw it then and not when you upgraded from 0.17 to 0.18? You might see this message when upgrading to 0.18 from 0.17. You said that you created indices and eventually it got stuck on .18 cluster, can you gist the settings / mappings for those indices that you created. I would like to try and simulate it here. -shay.banon On Tue, Nov 22, 2011 at 12:07 AM, Grant [grant@brewster.com](mailto:grant@brewster.com) wrote: > Today we attempted an upgrade from 0.17.6 to 0.18.4. We run a 4 node > cluster with lots of small indexes (roughly 0-800mb each) each with 2 > replicas, no sharding. We had about 3gb of data on each node at the > time. > > After a full restart, the cluster failed to recover and remained in a > red state until nodes would start to become unresponsive. This > happened as soon as the processes resident size equaled its JVM heap > size (2gb initially). Raising the heap to 8gb had no effect. ES would > just consume all the available heap then become unresponsive after > having done so. > > We eventually decided to re-index (starting with an empty data > directory on all nodes), still on 0.18.4, and discovered that we could > create about 10 (empty) indexes in rapid succession and then ES would > become unresponsive. It was also very slow to move out of red or > yellow states. > > Having lost most hope by this point, we downgraded to 0.17.6 and began > indexing again, and everything flew along in typical fashion. > > Anyone have any thoughts as to what gives? > > The only useful log messages I turned up was when we were trying to re- > index from scratch on 0.18.4... these streamed continuously: > > [2011-11-21 21:46:19,421][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,485][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,544][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,605][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,670][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,730][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,792][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,857][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,916][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:19,975][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,043][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,102][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,163][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,224][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,292][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,351][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-3-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,411][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-7-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,474][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,534][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] > [2011-11-21 21:46:20,594][WARN][cluster.metadata] [prod- > elasticsearch-r02] [contact\_documents-11-0] re-syncing mappings with > cluster state for types [[contact\_document]] ---
**Author:** ![Pierre\_Baillet\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierre_baillet_2/32/3063_2.png) [@Pierre\_Baillet\_2](https://discuss.elastic.co/u/Pierre_Baillet_2)\ **Post date:** [November 28, 2011, 1:48pm UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933/4 "2011-11-28T13:48:11Z")
Hi, On Nov 22, 8:56 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Hey, > > You should not see the "re-sync" message when indexing into a fresh > 0.18.4 cluster, are you sure you saw it then and not when you upgraded from > 0.17 to 0.18? You might see this message when upgrading to 0.18 from 0.17. Having the same problem even after our indices were recreated from scratch, I debugged the code in the MetaDataMappingService#refreshMapping on version 0.18.2 (864d95d1543fb712e2b1b3a11be5a5c935b93624) and found out that: - at line 127, mapper.mappingSource() and indexMetaData.mappings().get(type).source()) differ in my case. - once split by line at }, the diff of the two sources is: --- a 2011-11-28 14:32:54.000000000 +0100 +++ b 2011-11-28 14:33:05.000000000 +0100 @@ -16,8 +16,8 @@ } ,"location":{"type":"string","store":"yes","include\_in\_all":true} ,"name":{"type":"multi\_field","fields":{"name": {"type":"string","boost": 2.1,"store":"yes","omit\_term\_freq\_and\_positions":true,"include\_in\_all":true} -,"fuzzy": {"type":"string","analyzer":"fuzzy","omit\_term\_freq\_and\_positions":true,"include\_in\_all":false} ,"alphaSort": {"type":"string","analyzer":"alphasort","omit\_term\_freq\_and\_positions":true,"include\_in\_all":false} +,"fuzzy": {"type":"string","analyzer":"fuzzy","omit\_term\_freq\_and\_positions":true,"include\_in\_all":false} ,"autocomplete": {"type":"string","index\_analyzer":"ngram","search\_analyzer":"auto\_complete\_search\_analyser","include\_in\_all":false} } } As you can see, it's a simple reorder of the mapping that causes the difference and endless synchronize the mappings. The gist of the two original mappings is available at [Mapping table · GitHub](https://gist.github.com/1400435) . This issue eats 100% of the cpu of our master which make ugly graphs in our monitoring tool and made me worry a bit when I discovered the issue at first. Cheers, Pierre. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 28, 2011, 2:17pm UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933/5 "2011-11-28T14:17:20Z")
Hey, Yea, another user has hit that, with multi field mapping and multiple additional fields mapped: [Multi field mapper with more than one extra mapping can cause endless re-sync'ing of mapping between nodes · Issue #1487 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1487). It will be part of 0.18.5. -shay.banon On Mon, Nov 28, 2011 at 3:48 PM, Pierre Baillet [octplane@gmail.com](mailto:octplane@gmail.com) wrote: > Hi, > > On Nov 22, 8:56 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Hey, > > > > You should not see the "re-sync" message when indexing into a fresh > > 0.18.4 cluster, are you sure you saw it then and not when you upgraded > > from > > 0.17 to 0.18? You might see this message when upgrading to 0.18 from > > 0.17. > > Having the same problem even after our indices were recreated from > scratch, I debugged the code in the > MetaDataMappingService#refreshMapping on version 0.18.2 > (864d95d1543fb712e2b1b3a11be5a5c935b93624) and found out that: > > - at line 127, mapper.mappingSource() and > indexMetaData.mappings().get(type).source()) differ in my case. > - once split by line at }, the diff of the two sources is: > > --- a 2011-11-28 14:32:54.000000000 +0100 > +++ b 2011-11-28 14:33:05.000000000 +0100 > @@ -16,8 +16,8 @@ > } > ,"location":{"type":"string","store":"yes","include\_in\_all":true} > ,"name":{"type":"multi\_field","fields":{"name": > {"type":"string","boost": > > 2.1,"store":"yes","omit\_term\_freq\_and\_positions":true,"include\_in\_all":true} > -,"fuzzy": > > {"type":"string","analyzer":"fuzzy","omit\_term\_freq\_and\_positions":true,"include\_in\_all":false} > ,"alphaSort": > > {"type":"string","analyzer":"alphasort","omit\_term\_freq\_and\_positions":true,"include\_in\_all":false} > +,"fuzzy": > > {"type":"string","analyzer":"fuzzy","omit\_term\_freq\_and\_positions":true,"include\_in\_all":false} > ,"autocomplete": > > {"type":"string","index\_analyzer":"ngram","search\_analyzer":"auto\_complete\_search\_analyser","include\_in\_all":false} > } > } > > As you can see, it's a simple reorder of the mapping that causes the > difference and endless synchronize the mappings. > > The gist of the two original mappings is available at > [Mapping table · GitHub](https://gist.github.com/1400435) > . > > This issue eats 100% of the cpu of our master which make ugly graphs > in our monitoring tool and made me worry a bit when I discovered the > issue at first. > > Cheers, > Pierre. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:47am UTC](https://discuss.elastic.co/t/problems-with-0-18-x/5933/6 "2017-07-06T03:47:17Z")