# Weird behavior using norms with nested type **URL:** \ **Category:** Elasticsearch\ **Created:** [May 24, 2019, 3:16pm UTC](https://discuss.elastic.co/t/weird-behavior-using-norms-with-nested-type/182684 "2019-05-24T15:16:09Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![kailoofi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailoofi/32/46812_2.png) [@kailoofi](https://discuss.elastic.co/u/kailoofi)\ **Post date:** [May 24, 2019, 3:16pm UTC](https://discuss.elastic.co/t/weird-behavior-using-norms-with-nested-type/182684/1 "2019-05-24T15:16:09Z")
Hello, I am observing inexplicable behavior and looking to reconcile that with my understanding of norms and nested types. I am using ES 5.6 as of now and not sure if this can be observed in more recent versions. I have 2 mappings as shown below **Mapping 1 uses copy\_to to copy data from 1 object into another object which has a text multifield:** ``` "field1": { "properties": { "subfield1": { "type": "keyword", "index": false, "doc_values": false, "store": false, "copy_to": ["field2.subfield2"] } } } "field2": { "properties": { "subfield2": { "type": "keyword", "index": false, "doc_values": false, "store": false, "fields": { "textmultifield": { "type": "text", "index_options": "offsets", "norms": true, "store": false } } } } } ``` **Mapping 2 is similar except the the src object is of nested type:** ``` "nestedfield": { "type": "nested", "properties": { "subfield": { "type": "keyword", "index": false, "doc_values": false, "store": false, "copy_to": ["objectfield.subfield2"] } } } "objectfield": { "properties": { "subfield2": { "type": "keyword", "index": false, "doc_values": false, "store": false, "fields": { "textmultifield": { "type": "text", "index_options": "offsets", "norms": true, "store": false } } } } } ``` > **"\_all"** field is disabled completely and " **\_source"** is enabled for both mappings Since the norms is enabled only on the **textmultifield** and norms are disabled by default if the field is not indexed, I assumed that the norms would be the same for both mappings given the same dataset but in reality the disk usage of norms in the 2nd case is higher by a magnitude or more. I verified that the norms take a lot of space in the 2nd case by analyzing the disk storage utilized by the underlying index files. Can anyone help me in understanding the discrepancies between the 2 cases? It seems like there is an underlying detail about nested types that has been documented. Thanks in advance. ---
**Author:** ![kailoofi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailoofi/32/46812_2.png) [@kailoofi](https://discuss.elastic.co/u/kailoofi)\ **Post date:** [May 28, 2019, 8:40pm UTC](https://discuss.elastic.co/t/weird-behavior-using-norms-with-nested-type/182684/2 "2019-05-28T20:40:51Z")
> [@kailoofi](#): > > ot indexed, I assumed that the norms would be the same for both mappings given the same dataset but in reality the disk usage of norms in the 2nd case is higher by a magnitude or more. I verified that the norms take a lot of space in the 2nd case by analyzing the disk storage utilized by the underlying index files. > > Can anyone help me in understanding the discrepancies between the 2 cases? It seems like there is an underlying detail about nested type If anyone has experienced this or can explain this phenomenon, it would help me a lot. Thanks again. ---
**Author:** ![kailoofi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kailoofi/32/46812_2.png) [@kailoofi](https://discuss.elastic.co/u/kailoofi)\ **Post date:** [June 4, 2019, 5:38pm UTC](https://discuss.elastic.co/t/weird-behavior-using-norms-with-nested-type/182684/3 "2019-06-04T17:38:01Z")
I digged further into the code and noticed that norms were being created even when **omitNorms flag is set to true** in the fieldTypes in **DefaultIndexingChain.java** file in lucene. This is the code that essentially is invoked as part of DocsWriter during indexing from Elasticsearch. I chased this and found that the method **getOrAddField** tries to create a FieldInfo object in the 1st pass. By default this object has omitNorms to false. The method sets the indexOptions as specified in the fieldType on this newly created object but doesn't do the same for omitNorms. This effectively overrides this flag which creates issues down the line. Here's the code snippet for the method with the **fieldInfos.getOrAdd** call ``` private PerField getOrAddField(String name, IndexableFieldType fieldType, boolean invert) { // Make sure we have a PerField allocated final int hashPos = name.hashCode() & hashMask; PerField fp = fieldHash[hashPos]; while (fp != null && !fp.fieldInfo.name.equals(name)) { fp = fp.next; } if (fp == null) { // First time we are seeing this field in this segment FieldInfo fi = fieldInfos.getOrAdd(name); // Messy: must set this here because e.g. FreqProxTermsWriterPerField looks at the initial // IndexOptions to decide what arrays it must create). Then, we also must set it in // PerField.invert to allow for later downgrading of the index options: fi.setIndexOptions(fieldType.indexOptions()); fp = new PerField(fi, invert); ... ``` This issue is compounded when using nested types, there are internal fields in Elasticsearch like \_field\_names that are repeated multiple times in the top-level document with omitNorms set to true but I think due to this issue in Lucene 6.6.1, that flag is being overriden and norms are being computed and written to the disk unintentionally. Can some from the Elastic community confirm this? This is severely impacting our platform. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 2, 2019, 5:38pm UTC](https://discuss.elastic.co/t/weird-behavior-using-norms-with-nested-type/182684/4 "2019-07-02T17:38:04Z")
Closed on July 2, 2019, 5:38pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.