# How to take profit of compression for a Time Serie Data Stream **URL:** \ **Category:** Elasticsearch\ **Tags:** datastreams\ **Created:** [October 11, 2023, 11:28am UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811 "2023-10-11T11:28:53Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Mubolio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mubolio/32/121792_2.png) [@Mubolio](https://discuss.elastic.co/u/Mubolio)\ **Post date:** [October 11, 2023, 11:28am UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811/1 "2023-10-11T11:28:53Z")
Hello, Every hour I generate a bulk request with several documents into a data stream, each update sets a @timestamp field for all the bulk of documents. (the idea is to keep track of a source of data in time). The problem is that comparing each update from the previous there are several duplicated documents. To optimize the size of the datastream I thought that is was a good idea to use TSDS tool since looks that can acomplish aproximatelly a 70% of compression. When I run it, I don't see any compression going on, the size of the data stream is the same as I don't use Time Series. Also I observe that two updates(both updates add the same documents) has the double size of one update(it is not compressing). Does someone know how can I compress the duplicated fields from documents with different timestamp ? Thanks and best regards. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 11, 2023, 11:55am UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811/2 "2023-10-11T11:55:14Z")
What does your document looks like? Is it your data logs or traces? Also, can you share the template of your TSDS? ---
**Author:** ![Mubolio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mubolio/32/121792_2.png) [@Mubolio](https://discuss.elastic.co/u/Mubolio)\ **Post date:** [October 11, 2023, 12:06pm UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811/3 "2023-10-11T12:06:09Z")
The documents are time stamped metrics. (I guess not the most common use of ES?) Each document contains metadata related to a file and it describes properties linked to that file. (the file is stored in a remote hard drive that it is update continuosly). An example of a document: ```auto { "filename": "path_filename_with_metadata", "metadata1":"m1", "metadata2":"m2", "metadata3":"m3", "@timestamp": "2023-10-11T13:47:05.000Z" } ``` filename field is unique, it will not be found more than one in one update. That's the field that I set as "time\_series\_dimension": true in the mapping of the index\_template. This is the index\_template: ```auto { "template": { "settings": { "index": { "lifecycle": { "name": "policy" }, "mode": "time_series", "codec": "best_compression", "routing": { "allocation": { "include": { "_tier_preference": "data_hot" } } }, "routing_path": [ "file_name" ], "time_series": { "end_time": "2023-10-11T14:03:06.000Z", "start_time": "2023-10-11T10:03:06.000Z" } } }, "mappings": { "properties": { "file_name": { "type": "keyword", "time_series_dimension": true } } }, "aliases": {} } } ``` Thanks. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [October 11, 2023, 12:30pm UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811/4 "2023-10-11T12:30:51Z")
Any reason of why you didn't set `time_series_dimension` for the `metadataX` fields? I don't think you will see much difference in compression comparing with datastream with just one field as `time_series_dimesions`, specially if you were already using the `best_compression` codec in the normal data stream. ---
**Author:** ![Mubolio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mubolio/32/121792_2.png) [@Mubolio](https://discuss.elastic.co/u/Mubolio)\ **Post date:** [October 11, 2023, 1:05pm UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811/5 "2023-10-11T13:05:26Z")
I have added more fields as time\_series\_dimension fields. I don't see any compression when I update 3 times the bulk, the datastream takes 3x the size of the first bulk update. Let's say that the TSDS has this content, just as an example both update has the same documents: (First update) doc1 with @timestamp=1 doc2 with @timestamp=1 ... docn with @timestamp=1 (Second update) doc1 with @timestamp=2 doc2 with @timestamp=2 ... docn with @timestamp=2 Considering that docs are the same in both updates but the only difference will be the timestamp field (we can say that in the datastream each doc is one time duplicated besides the timstamp field) and all fields in the docs are dimensions. Being this case should I see any compression going on? (I mean the compression that takes profit of duplicated fields after more than one bulk update, it is of course compressing a lot just after the firtst bulk update, since the total docs size is way less than the sum of each individual doc). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 8, 2023, 1:05pm UTC](https://discuss.elastic.co/t/how-to-take-profit-of-compression-for-a-time-serie-data-stream/344811/6 "2023-11-08T13:05:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.