# Elasticsearch Data Streams: Update Strategies, Concerns, and Alternatives **URL:** \ **Category:** Elasticsearch\ **Tags:** datastreams\ **Created:** [January 8, 2024, 7:12am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546 "2024-01-08T07:12:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jainesh\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jainesh_singh/32/119545_2.png) [@jainesh\_singh](https://discuss.elastic.co/u/jainesh_singh)\ **Post date:** [January 8, 2024, 7:12am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/1 "2024-01-08T07:12:56Z")
Hi Team, I am stuck and need your help!! **UseCase:** I am using elasticsearch where i am storing activities in a data stream. I want to perform update operation on this Data stream. There is time based range queries that are fired to fetch data and the number of records is quite high, nearly 1-2 crore. **Problem:** I am currently using logstash for other processes but since logstash does not out of the box provide a way to update data stream it is not fitting my use case as i would need to update activities On reading i found out that elasticsearch supports updateby query for updating data streams Reference : [How to update data stream?](https://www.elastic.co/guide/en/elasticsearch/reference/current/use-a-data-stream.html#update-delete-docs-in-a-backing-index) **Ask:** 1. Can i use this feature via elasticsearch client in my code, i.e. I will not use logstash to update the Data stream, is it a correct way to use it? 2. Could it happen that going forward ES could remove this option of updating data stream via api? 3. Is there any alternative way in logstash or in general I could solve this use case? I just want faster retrieval of data in a time based, large data set of data.. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2024, 8:40am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/2 "2024-01-08T08:40:29Z")
> [@jainesh\_singh](#): > > I am currently using logstash for other processes but since logstash does not out of the box provide a way to update data stream it is not fitting my use case as i would need to update activities > [@jainesh\_singh](#): > > Could it happen that going forward ES could remove this option of updating data stream via api? Data streams are [already being optimised for immutable data](https://www.elastic.co/guide/en/elasticsearch/reference/8.11/data-streams.html), so if you need to update data I would recommend against using data streams. > [@jainesh\_singh](#): > > Is there any alternative way in logstash or in general I could solve this use case? It would help if you told us a bit about the use case. - What kind of data is it? - How long do you keep the data? - How often do you update the data? - How do you update the data? Are you replacing with a new record or just changing a few fields? - Do you know the timestamp associated with the initial event when you perform the update? It would also help if you indicated which version of Elasticsearch you are using. ---
**Author:** ![jainesh\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jainesh_singh/32/119545_2.png) [@jainesh\_singh](https://discuss.elastic.co/u/jainesh_singh)\ **Post date:** [January 8, 2024, 8:56am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/3 "2024-01-08T08:56:09Z")
Hi Christian, Please find answer to your questions below. - What kind of data is it? It is reporting data so it contains fields like {activityId, fileID, owner, activityTime, .. etc} - How long do you keep the data? We keep the data for 8 months - How often do you update the data? Quite frequently - How do you update the data? Are you replacing with a new record or just changing a few fields? We are just changing a few fields, in all 4 fields out of nearly 20 fields.. - Do you know the timestamp associated with the initial event when you perform the update? We know the timestamp of the initial/previous/to be updated activity which is activityTime as referred above in question1. This time indicates when the activity was performed. It would also help if you indicated which version of Elasticsearch you are using. Certainly, we are using Elasticsearch version 8.6.0 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2024, 8:59am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/4 "2024-01-08T08:59:29Z")
In that case I would recommend using traditional time-based indices where the date is part of the index name and send data to the correct index based on the known timestamp. Depending on data volumes you may want to use daily or monthly indices with a reasonable number of primary shards (aim for a shard size of a few tens of GB). ---
**Author:** ![jainesh\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jainesh_singh/32/119545_2.png) [@jainesh\_singh](https://discuss.elastic.co/u/jainesh_singh)\ **Post date:** [January 10, 2024, 5:45am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/5 "2024-01-10T05:45:01Z")
Hey Christian, Thanks for your help. I feel the solution you suggested would work for me. Using your suggested approach, I am facing certain challenges in Logstash implementation. Could you share some light on : > [@Updating Elasticsearch Indices conditionally when referring to 2 database table](https://discuss.elastic.co/t/updating-elasticsearch-indices-conditionally-when-referring-to-2-database-table/350663): > > Description: We have two SQL tables: FileDetail for storing file details and FileUserActivity for file activities. Using Logstash, we're indexing data into Elasticsearch with a flat index approach, combining file details and activities in a single document. However, when file details change in the FileDetail table (e.g., file owner modification), we need a solution to update all previous Elasticsearch indices related to that file. Database Tables: FileDetail Table: Columns: fileid, ..., o… ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 10, 2024, 6:32am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/6 "2024-01-10T06:32:19Z")
As the discussion is now centered around how to handle this in Logstash, lets continue the discussion in the other thread. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 7, 2024, 6:32am UTC](https://discuss.elastic.co/t/elasticsearch-data-streams-update-strategies-concerns-and-alternatives/350546/7 "2024-02-07T06:32:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.