# Optimize for minimal storage space with many tiny documents **URL:** \ **Category:** Elasticsearch\ **Created:** [November 29, 2015, 12:44am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811 "2015-11-29T00:44:09Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Andre\_Hansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_hansel/32/934_2.png) [@Andre\_Hansel](https://discuss.elastic.co/u/Andre_Hansel)\ **Post date:** [November 29, 2015, 12:44am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/1 "2015-11-29T00:44:09Z")
I want to store a lot of data points to run aggregations on them. Here's my index definition: ``` mappings: { datapoint: { properties: { timestamp: { type: 'date', format: 'date_time'}, value: { type: 'float', index: 'no', store: false, doc_values: true }, metric_name: { type: 'string', index: 'not_analyzed', store: false, doc_values: true } }, _all: { enabled: false}, _source: { enabled: false} } } ``` The two queries that I'd like to run on this data are: "For `timestamp` in range `a-b` and `metric_name` c, what is the sum of `value`?" and "Which `metric_name`s do we have?" There are 800 different metric names, each one about 80 bytes long. Since the cardinality is so low, I was hoping for a good compression ratio. There are 8 byte per document of real (incompressible) data, plus the ID which is auto-generated by Elasticsearch. Right now I am looking at about 3,5 GB for 100M documents of this type, i.e. about 37 bytes per document. Since I am creating a few million new datapoints per day, I'd like to know if I can get the storage requirements further down. Is there anything I can optimize further? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 29, 2015, 1:24am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/2 "2015-11-29T01:24:55Z")
What version are you on? ---
**Author:** ![AndreKR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrekr/32/6233_2.png) [@AndreKR](https://discuss.elastic.co/u/AndreKR)\ **Post date:** [November 29, 2015, 10:57am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/3 "2015-11-29T10:57:05Z")
The machine I was testing with has 1.7.3, because I wanted the Sense plugin for easier testing. I could upgrade to 2.0. ---
**Author:** ![AndreKR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrekr/32/6233_2.png) [@AndreKR](https://discuss.elastic.co/u/AndreKR)\ **Post date:** [November 30, 2015, 7:48pm UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/4 "2015-11-30T19:48:00Z")
I upgraded to 2.1 and set `codec: best_compression` and now I am at 25 bytes per document. That's already better but of course I'd still take suggestions to go down even further. ---
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [March 14, 2016, 11:45am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/5 "2016-03-14T11:45:27Z")
Can you store the metric-name in a database and then reference [metric.id](http://metric.id) which will be a 8byte integer ? ---
**Author:** ![AndreKR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrekr/32/6233_2.png) [@AndreKR](https://discuss.elastic.co/u/AndreKR)\ **Post date:** [March 18, 2016, 2:19am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/6 "2016-03-18T02:19:52Z")
I have only about 1000 unique metric names. If replacing them with an integer values helps, there must be something seriously wrong with Elasticsearch's index format. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 18, 2016, 5:43am UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/7 "2016-03-18T05:43:45Z")
Are you letting Elasticsearch assign the IDs of the documents? If you know your data very well, you might be able to generate a unique key at the application level that is more compact and save space that way. When designing a key, it is worth reading [the advice in this blog post](http://blog.mikemccandless.com/2014/05/choosing-fast-unique-identifier-uuid.html). ---
**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\ **Post date:** [March 18, 2016, 5:48pm UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/8 "2016-03-18T17:48:10Z")
Make sure it's a short integer then. Also shorten field names to 1 character and report back if the size shrinked. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:07pm UTC](https://discuss.elastic.co/t/optimize-for-minimal-storage-space-with-many-tiny-documents/35811/9 "2017-07-05T23:07:05Z")