# How does elasticsearch store repeated values across documents?

**URL:** <https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143>\
**Category:** Elasticsearch\
**Created:** [March 1, 2018, 7:28pm UTC](https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143 "2018-03-01T19:28:26Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![JonathanAaron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathanaaron/32/24405_2.png) [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\
**Post date:** [March 1, 2018, 7:28pm UTC](https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143/1 "2018-03-01T19:28:26Z")

</div>

Say I have a document

`{"name":"first","class": "Type-A"}`

Then another document

`{"name":"second","class": "Type-A"}`

Does it store Type-A twice or does it reference repeated values. Lets say I have a million docs like this referring to Type-A? Do ES optimize this somehow?

Thanks,

Jonathan

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 2, 2018, 1:06am UTC](https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143/2 "2018-03-02T01:06:58Z")

</div>

Two things. Elasticsearch builds an inverted index out of that basically, the inverted index part will contain something like:

```auto
Type-A: 1, 2

```

Where 1 and 2 are the documents id. That's schematic as other things are also added.

But, elasticsearch will also add a stored field named `_source` which will contain:

```auto
{"name":"first","class": "Type-A"}

```

And

```auto
{"name":"second","class": "Type-A"}

```

As is. But this is compressed by default.

Normally, you don't really have to think about all this.

> Do ES optimize this somehow?

As yes, ES optimized as much as possible all that.

---

<div class="post-metadata">

**Author:** ![JonathanAaron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jonathanaaron/32/24405_2.png) [@JonathanAaron](https://discuss.elastic.co/u/JonathanAaron)\
**Post date:** [March 2, 2018, 4:11pm UTC](https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143/3 "2018-03-02T16:11:01Z")

</div>

Thanks for the response. My team is facing space issues and we want to add another attribute to our documents to query by, but that involves updating billions of documents. So I'm trying to estimate the space cost. Seems the inverted index space would be trivial. Anyway to figure out how elastic compresses things?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 2, 2018, 5:15pm UTC](https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143/4 "2018-03-02T17:15:22Z")

</div>

> My team is facing space issues

Well. Really often the cost of complexity and the tradeoffs are much bigger than buying new hardware (disks). But I can't tell for you.

> Anyway to figure out how elastic compresses things?

More about compression here: [Index Modules | Elasticsearch Reference [6.2] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/index-modules.html#index-codec)

You can read this section which gives a lot of advices:

> **[Tune for disk usage | Elasticsearch Reference \[6.2\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/tune-for-disk-usage.html)**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2018, 5:15pm UTC](https://discuss.elastic.co/t/how-does-elasticsearch-store-repeated-values-across-documents/122143/5 "2018-03-30T17:15:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.