# ES 6 sparse docs and index.mapping.total\_fields.limit

**URL:** <https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196>\
**Category:** Elasticsearch\
**Created:** [July 14, 2017, 1:21pm UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196 "2017-07-14T13:21:59Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [July 14, 2017, 1:21pm UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/1 "2017-07-14T13:21:59Z")

</div>

I'm currently trying to understand how to map http request parameters into Elasticsearch.

Ideally, I'd have one field per query parameter

```auto
parameters:
  foo: "bar"
  baz: "qux"
  q: "search query"

```

This would allow me to easily analyze the search queries by performing a terms aggregation on `parameters.q`.

But as I've learned, this leads to [sparse documents](https://www.elastic.co/guide/en/elasticsearch/reference/current/general-recommendations.html#sparsity) which are bad. It's also possible that you hit the [total fields limit](https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping.html#mapping-limit-settings) of 1000 fields by mapping parameters that way.

But as Lucene 7/Elasticsearch 6 will come with better support for sparse documents ([https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released#sparse-doc-values](https://www.elastic.co/blog/elasticsearch-6-0-0-alpha1-released#sparse-doc-values)), does the recommendation against sparse documents still hold true? Will the field limit of 1000 still remain in ES 6? How would you map request parameters to Elasticsearch now and in ES 6?

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [July 17, 2017, 8:25am UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/2 "2017-07-17T08:25:12Z")

</div>

@warkolm Hey bud, could you help me out here?

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [July 24, 2017, 2:21pm UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/3 "2017-07-24T14:21:35Z")

</div>

@spinscale do you have any information on this?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [July 25, 2017, 10:57am UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/4 "2017-07-25T10:57:46Z")

</div>

Hey,

where as the sparse documents issue has been improved dramatically, you still run into the issue of exploding field mapping and huge cluster state. Have you thought of changing your model to a `nested` type with elements like

```auto
parameters: [
  { "key":"foo", "value":"bar"}
  { "key":"spam", "value":"eggs"}
]

```

to prevent mapping explosion?

---

<div class="post-metadata">

**Author:** ![Amit\_Ripshtos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amit_ripshtos/32/40400_2.png) [@Amit\_Ripshtos](https://discuss.elastic.co/u/Amit_Ripshtos)\
**Post date:** [July 25, 2017, 11:19am UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/5 "2017-07-25T11:19:14Z")

</div>

As a person who shares almost the same problem (each user has it's own "schema"), this nested trick is not helping much since the searches/aggregations become wrong because of it, and it's a lot nicer to use kibana without this schema (because for example in kibana you need to show key "foo" and "spam" in same graph)

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [July 25, 2017, 1:55pm UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/6 "2017-07-25T13:55:12Z")

</div>

> [@spinscale](#):
>
> where as the sparse documents issue has been improved dramatically, you still run into the issue of exploding field mapping and huge cluster state

That's an important information for me, thx. Seems like the nested approach is the only way to go performance-wise.

But the problem is that i can't do a terms aggregation of the foo parameter, right? Or is there a workaround for that?

Maybe I'll add a configuration option for a explicit whitelist of parameters which should be converted to the non nested version and use nested parameters by default...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 22, 2017, 1:55pm UTC](https://discuss.elastic.co/t/es-6-sparse-docs-and-index-mapping-total-fields-limit/93196/7 "2017-08-22T13:55:17Z")

</div>

Closed on August 22, 2017, 1:55pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.