# \#datastreams **URL:** https://discuss.elastic.co/tag/datastreams/85.md [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) --- ## [Hot/Warm/Cold phases being ignored. Data goes directly to Cold](https://discuss.elastic.co/t/hot-warm-cold-phases-being-ignored-data-goes-directly-to-cold/389910)
**Author:** [@Alberto\_Martinez](https://discuss.elastic.co/u/Alberto_Martinez)\ **Replies:** 7\ **Last updated:** [September 15, 2026, 12:41pm UTC](https://discuss.elastic.co/t/hot-warm-cold-phases-being-ignored-data-goes-directly-to-cold/389910 "2026-09-15T12:41:27Z")
Hello, As the title says, all my streams share a common ILM policy but somehow it is ignored. All nodes are capable of hot/warm/cold. It is a recent setup and we don't have separate roles yet. Any pointer to wha… --- ## [How to reindex when source indices are continuously receiving data?](https://discuss.elastic.co/t/how-to-reindex-when-source-indices-are-continuously-receiving-data/385685)
**Author:** [@yogesh119905](https://discuss.elastic.co/u/yogesh119905)\ **Replies:** 10\ **Last updated:** [August 3, 2026, 7:41pm UTC](https://discuss.elastic.co/t/how-to-reindex-when-source-indices-are-continuously-receiving-data/385685 "2026-08-03T19:41:48Z")
Title: How to reindex when source indices are continuously receiving data? Description: I’m trying to perform a reindex operation in Elasticsearch where the source indices are continuously receiving new data. Scenari… --- ## [Migrate unmanaged index to data stream](https://discuss.elastic.co/t/migrate-unmanaged-index-to-data-stream/387725)
**Author:** [@Ts\_P](https://discuss.elastic.co/u/Ts_P)\ **Replies:** 2\ **Last updated:** [July 10, 2026, 8:24am UTC](https://discuss.elastic.co/t/migrate-unmanaged-index-to-data-stream/387725 "2026-07-10T08:24:58Z")
Hi, I have an index that receives a large amount of data. It is currently 250 GB in size (Elasticsearch 9.4.2), and data is continuously being ingested into it. I want to migrate it to a data stream. I also want to ap… --- ## [The proper way to reindex a data stream index](https://discuss.elastic.co/t/the-proper-way-to-reindex-a-data-stream-index/371102)
**Author:** [@Dave\_Houser](https://discuss.elastic.co/u/Dave_Houser)\ **Replies:** 2\ **Last updated:** [June 25, 2026, 2:03pm UTC](https://discuss.elastic.co/t/the-proper-way-to-reindex-a-data-stream-index/371102 "2026-06-25T14:03:56Z")
I have run into an issue with mappings in an data stream index. Can someone help me out? I am getting data type conflicts between the new mapping and old mapping of some fields. What I want to do is just remove the mappi… --- ## [Elasticsearch Serverless + Vector Search + Data Streams / Time Series Tradeoffs](https://discuss.elastic.co/t/elasticsearch-serverless-vector-search-data-streams-time-series-tradeoffs/386275)
**Author:** [@Januka\_Samaranayake](https://discuss.elastic.co/u/Januka_Samaranayake)\ **Replies:** 3\ **Last updated:** [May 12, 2026, 8:16am UTC](https://discuss.elastic.co/t/elasticsearch-serverless-vector-search-data-streams-time-series-tradeoffs/386275 "2026-05-12T08:16:25Z")
Hello everyone, I am trying to better understand the tradeoffs between Elasticsearch Serverless, Data Streams, TSDS (index.mode=time\_series), and future vector search workloads. Current Situation We currently have arou… --- ## [A major issue with cluster state handling and persistent tasks cancellation](https://discuss.elastic.co/t/a-major-issue-with-cluster-state-handling-and-persistent-tasks-cancellation/386014)
**Author:** [@sherman81](https://discuss.elastic.co/u/sherman81)\ **Replies:** 7\ **Last updated:** [April 27, 2026, 8:15am UTC](https://discuss.elastic.co/t/a-major-issue-with-cluster-state-handling-and-persistent-tasks-cancellation/386014 "2026-04-27T08:15:53Z")
We are using ES 9.3.0. Our cluster has many data streams and indices. The cluster state is ~350 MB (compressed on disk) under normal conditions. I mistakenly scheduled a large number of downsampling tasks for historica… --- ## [Docker logs integration with Fleet to a datastream - my experience](https://discuss.elastic.co/t/docker-logs-integration-with-fleet-to-a-datastream-my-experience/385549)
**Author:** [@dot-mike](https://discuss.elastic.co/u/dot-mike)\ **Replies:** 0\ **Last updated:** [March 20, 2026, 4:43pm UTC](https://discuss.elastic.co/t/docker-logs-integration-with-fleet-to-a-datastream-my-experience/385549 "2026-03-20T16:43:07Z")
Hi community, I'm sharing my experience here for ingesting docker logs with fleet agent-setup. My goal is to ingest docker logs for a specific application running on on-premises on a specific system to a custom datastre… --- ## [Struggling to set up multiple namespaces for a single Beats data stream](https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212)
**Author:** [@dot-mike](https://discuss.elastic.co/u/dot-mike)\ **Replies:** 2\ **Last updated:** [November 4, 2025, 4:27pm UTC](https://discuss.elastic.co/t/struggling-to-set-up-multiple-namespaces-for-a-single-beats-data-stream/383212 "2025-11-04T16:27:54Z")
Dear community In Elastic Fleet you can set a namespace for an integration so you can have multiple data streams for a single type of index template without having to duplicate settings (ILM policy, index template, comp… --- ## [How to reindex a data stream into another data stream?](https://discuss.elastic.co/t/how-to-reindex-a-data-stream-into-another-data-stream/383131)
**Author:** [@dot-mike](https://discuss.elastic.co/u/dot-mike)\ **Replies:** 2\ **Last updated:** [November 3, 2025, 2:02pm UTC](https://discuss.elastic.co/t/how-to-reindex-a-data-stream-into-another-data-stream/383131 "2025-11-03T14:02:06Z")
Hi community, I’m trying to reindex data from one data stream to another. Basically, I want data from this data stream logs-syslog.myproduct.logs to go into logs-syslog.myproduct\_logs-default The issue is that when th… --- ## [Historical (Past) Data Ingestion in Time Series Data Stream](https://discuss.elastic.co/t/historical-past-data-ingestion-in-time-series-data-stream/379584)
**Author:** [@rubayetahmed314](https://discuss.elastic.co/u/rubayetahmed314)\ **Replies:** 14\ **Last updated:** [June 29, 2025, 6:04pm UTC](https://discuss.elastic.co/t/historical-past-data-ingestion-in-time-series-data-stream/379584 "2025-06-29T18:04:34Z")
Is it possible to insert historical (past) data into a Time Series Data Stream (TSDS) of Elasticsearch? To be more clear, suppose, I want to ingest NYC Taxi Trip Data from Year 2009 to 2024 in a time-series data stream. … --- ## [Where do OpenTelemetry logs end up in Elasticsearch](https://discuss.elastic.co/t/where-do-opentelemetry-logs-end-up-in-elasticsearch/379554)
**Author:** [@flalar](https://discuss.elastic.co/u/flalar)\ **Replies:** 0\ **Last updated:** [June 27, 2025, 7:35am UTC](https://discuss.elastic.co/t/where-do-opentelemetry-logs-end-up-in-elasticsearch/379554 "2025-06-27T07:35:45Z")
Hi all, I'm currently experimenting with using OpenTelemetry to send logs into Elasticsearch and trying to understand exactly where those logs end up once they arrive. So far, I see that metrics and traces follow somew… --- ## [Data mismatches happening while sending data to Elastic Search index using pyspark](https://discuss.elastic.co/t/data-mismatches-happening-while-sending-data-to-elastic-search-index-using-pyspark/377188)
**Author:** [@yolo1](https://discuss.elastic.co/u/yolo1)\ **Replies:** 5\ **Last updated:** [May 19, 2025, 10:24am UTC](https://discuss.elastic.co/t/data-mismatches-happening-while-sending-data-to-elastic-search-index-using-pyspark/377188 "2025-05-19T10:24:40Z")
Any idea why data sent through df.write. in pyspark the data doesn't match correctly . in the backend the data is correct. --- ## [Indexing rate for data streams](https://discuss.elastic.co/t/indexing-rate-for-data-streams/376566)
**Author:** [@Dr00py](https://discuss.elastic.co/u/Dr00py)\ **Replies:** 3\ **Last updated:** [April 2, 2025, 4:50pm UTC](https://discuss.elastic.co/t/indexing-rate-for-data-streams/376566 "2025-04-02T16:50:17Z")
I want to calculate the indexing speed of documents in primary shards in my data streams. When monitoring is enabled, I can see the index rate for each index. But I can't do this for any data stream from my clusters. I … --- ## [How to change a template of integration stream?](https://discuss.elastic.co/t/how-to-change-a-template-of-integration-stream/376005)
**Author:** [@111238](https://discuss.elastic.co/u/111238)\ **Replies:** 0\ **Last updated:** [March 17, 2025, 11:19am UTC](https://discuss.elastic.co/t/how-to-change-a-template-of-integration-stream/376005 "2025-03-17T11:19:04Z")
Hi there. I use custom UDP integration. And it creates a datastream with seemingly hardcoded default "logs" template. Creating another one with a higher priority doesn't help. How the heck do I change it? Creating a… --- ## [What's the relation between filebeat, index lifecycle policies and index templates?](https://discuss.elastic.co/t/whats-the-relation-between-filebeat-index-lifecycle-policies-and-index-templates/375455)
**Author:** [@jacek.bilski](https://discuss.elastic.co/u/jacek.bilski)\ **Replies:** 0\ **Last updated:** [March 5, 2025, 12:12pm UTC](https://discuss.elastic.co/t/whats-the-relation-between-filebeat-index-lifecycle-policies-and-index-templates/375455 "2025-03-05T12:12:12Z")
We're using Elastic 7.17. What I'm trying to accomplish is: I want to ship logs using filebeat so that they land in a datastream of a specific name and with a specific lifecycle policy. The goal is to separate data from … --- ## [Large log data indexing best practices (datastreams?)](https://discuss.elastic.co/t/large-log-data-indexing-best-practices-datastreams/374780)
**Author:** [@AnthonyKeydel](https://discuss.elastic.co/u/AnthonyKeydel)\ **Replies:** 2\ **Last updated:** [February 19, 2025, 8:59pm UTC](https://discuss.elastic.co/t/large-log-data-indexing-best-practices-datastreams/374780 "2025-02-19T20:59:38Z")
Hi there! I've been working to refine my Elastic instance for months now, and have gotten a bit tangled in the details. :face\_with\_spiral\_eyes: Before promoting the PoC into a live Production environment, I could use a… --- ## [Index template alias not being applied to backing indexes of datastream](https://discuss.elastic.co/t/index-template-alias-not-being-applied-to-backing-indexes-of-datastream/374127)
**Author:** [@robsonhermes](https://discuss.elastic.co/u/robsonhermes)\ **Replies:** 7\ **Last updated:** [February 7, 2025, 11:28am UTC](https://discuss.elastic.co/t/index-template-alias-not-being-applied-to-backing-indexes-of-datastream/374127 "2025-02-07T11:28:15Z")
Elastic version 8.17.1 Context: multiple systems logging to a datastream, so we want to create one dataview in Kiabana for each system. Infra provisioned via ansible, and our elastic/kibana objects/configs provisioned v… --- ## [Grouping Data Streams Based on Privilege Levels in UI](https://discuss.elastic.co/t/grouping-data-streams-based-on-privilege-levels-in-ui/373938)
**Author:** [@Akshraj\_Chavda](https://discuss.elastic.co/u/Akshraj_Chavda)\ **Replies:** 0\ **Last updated:** [January 31, 2025, 1:09pm UTC](https://discuss.elastic.co/t/grouping-data-streams-based-on-privilege-levels-in-ui/373938 "2025-01-31T13:09:57Z")
I have three data streams: data\_stream\_one data\_stream\_two data\_stream\_three And a single data collection input: CEL. Privilege Levels: Level C → Can access data\_stream\_one Level B → Can access data\_stream\_one and d… --- ## [Snapshot restore data stream problems](https://discuss.elastic.co/t/snapshot-restore-data-stream-problems/372170)
**Author:** [@Dave\_Houser](https://discuss.elastic.co/u/Dave_Houser)\ **Replies:** 0\ **Last updated:** [December 18, 2024, 2:48pm UTC](https://discuss.elastic.co/t/snapshot-restore-data-stream-problems/372170 "2024-12-18T14:48:02Z")
Running 8.13 I am working on restoring a data stream from a snapshot. I am doing this in a surgical way as I need to be able to write to indexes in snapshots while the restore of all other indexes happens in the backgro… --- ## [Restore Snapshot while writing to indexes/data streams?](https://discuss.elastic.co/t/restore-snapshot-while-writing-to-indexes-data-streams/371407)
**Author:** [@Dave\_Houser](https://discuss.elastic.co/u/Dave_Houser)\ **Replies:** 12\ **Last updated:** [December 16, 2024, 8:51pm UTC](https://discuss.elastic.co/t/restore-snapshot-while-writing-to-indexes-data-streams/371407 "2024-12-16T20:51:50Z")
I need to put together a DR plan for our elastic system. I have already tested the snapshot restore process, and it works. However, my process is the following: Adjust cluster settings to allow action.destructive\_requi… --- ## [Way to move many indexes to a data stream backing index?](https://discuss.elastic.co/t/way-to-move-many-indexes-to-a-data-stream-backing-index/372030)
**Author:** [@Dave\_Houser](https://discuss.elastic.co/u/Dave_Houser)\ **Replies:** 0\ **Last updated:** [December 16, 2024, 3:36pm UTC](https://discuss.elastic.co/t/way-to-move-many-indexes-to-a-data-stream-backing-index/372030 "2024-12-16T15:36:56Z")
Hello, I have been working on figuring out a process to restore some of a snapshot, start writing to it and allow our frontend to read from it. Then restore the rest of the snapshot. I think I have a good process. Howe… --- ## [Reindexing a datastream after changed mappings : best practice?](https://discuss.elastic.co/t/reindexing-a-datastream-after-changed-mappings-best-practice/370275)
**Author:** [@gpineda\_dev](https://discuss.elastic.co/u/gpineda_dev)\ **Replies:** 0\ **Last updated:** [November 10, 2024, 1:04am UTC](https://discuss.elastic.co/t/reindexing-a-datastream-after-changed-mappings-best-practice/370275 "2024-11-10T01:04:13Z")
Before writing this post, I started by gathering available information on this platform, official documentation, related articles. Examples to reindex basic indices are easy to find but when it comes to datastream, then… --- ## [Ingesting old data to a TSDS - Problems?](https://discuss.elastic.co/t/ingesting-old-data-to-a-tsds-problems/369174)
**Author:** [@deedubbs](https://discuss.elastic.co/u/deedubbs)\ **Replies:** 0\ **Last updated:** [October 21, 2024, 5:53pm UTC](https://discuss.elastic.co/t/ingesting-old-data-to-a-tsds-problems/369174 "2024-10-21T17:53:45Z")
We are about to stand up a time series datastream for some metrics data that we've been tracking via multiple indices. One of the notes on the TSDS documentation page suggests that you should only use a TSDS "if you typi… --- ## [Rollover action during a reindex](https://discuss.elastic.co/t/rollover-action-during-a-reindex/366821)
**Author:** [@brandon.n](https://discuss.elastic.co/u/brandon.n)\ **Replies:** 4\ **Last updated:** [September 19, 2024, 8:39pm UTC](https://discuss.elastic.co/t/rollover-action-during-a-reindex/366821 "2024-09-19T20:39:54Z")
Hello, Currently working on reindexing an old index with a massive 170ishGB single primary shard into a new datastream index, with an ILM policy of 30GB/30D. We configured for 8 primary, 1 replica, but currently the ro… --- ## [Updating the datastream from logstash](https://discuss.elastic.co/t/updating-the-datastream-from-logstash/366446)
**Author:** [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\ **Replies:** 1\ **Last updated:** [September 12, 2024, 6:13am UTC](https://discuss.elastic.co/t/updating-the-datastream-from-logstash/366446 "2024-09-12T06:13:35Z")
Hi Team, we are trying to lookup the data in one datastream with anothe data in another datastream based on a key using logstash elasticseach input plugin and elasticsearch filter as shown in the sample config below. M… --- ## [Handling different flows in data streams](https://discuss.elastic.co/t/handling-different-flows-in-data-streams/365342)
**Author:** [@Zain\_Ul\_Abideen](https://discuss.elastic.co/u/Zain_Ul_Abideen)\ **Replies:** 1\ **Last updated:** [August 22, 2024, 7:36am UTC](https://discuss.elastic.co/t/handling-different-flows-in-data-streams/365342 "2024-08-22T07:36:13Z")
Hello, I have been using data streams for my business flow logs, each flow has the different log structure. I have couple of solutions in mind: Having same structure for each flow and keep stringified version of log o… --- ## [Documents will not be found with querry, even if querry and mapping is correctand documents contain the fields](https://discuss.elastic.co/t/documents-will-not-be-found-with-querry-even-if-querry-and-mapping-is-correctand-documents-contain-the-fields/363528)
**Author:** [@felix.maier](https://discuss.elastic.co/u/felix.maier)\ **Replies:** 0\ **Last updated:** [July 22, 2024, 7:59am UTC](https://discuss.elastic.co/t/documents-will-not-be-found-with-querry-even-if-querry-and-mapping-is-correctand-documents-contain-the-fields/363528 "2024-07-22T07:59:26Z")
Hello, I have a Problem to find Documents in a Data Stream. Every Document has the fileds "received\_from" With an Update from the Filebeats, the filed "host.hostname" was added. Bothe of the filds hold the same string, a… --- ## [Applying ILM for backing indices of Elastic Integrations' datastreams](https://discuss.elastic.co/t/applying-ilm-for-backing-indices-of-elastic-integrations-datastreams/363533)
**Author:** [@An\_Hoang](https://discuss.elastic.co/u/An_Hoang)\ **Replies:** 0\ **Last updated:** [July 22, 2024, 8:34am UTC](https://discuss.elastic.co/t/applying-ilm-for-backing-indices-of-elastic-integrations-datastreams/363533 "2024-07-22T08:34:24Z")
Hello everyone, I am trying to apply the ILM for the backing indices of Elastic Integrations' datastreams. I see that we have a separate built-in Lifecycle of datastream to set the retention of data. The problem is I wa… --- ## [Kibana produces data view incrementally](https://discuss.elastic.co/t/kibana-produces-data-view-incrementally/361402)
**Author:** [@iamp3](https://discuss.elastic.co/u/iamp3)\ **Replies:** 3\ **Last updated:** [June 17, 2024, 9:43am UTC](https://discuss.elastic.co/t/kibana-produces-data-view-incrementally/361402 "2024-06-17T09:43:10Z")
Hi, It looks like smth block search/view of metric data in real-time mode (from otel-collectors) and produces this data incrementally. For example, on 12 June I could check data till 01:00, but today at 13 June I could s… --- ## [How to rollover data stream during reindex](https://discuss.elastic.co/t/how-to-rollover-data-stream-during-reindex/359765)
**Author:** [@jacob.k](https://discuss.elastic.co/u/jacob.k)\ **Replies:** 1\ **Last updated:** [May 19, 2024, 3:42pm UTC](https://discuss.elastic.co/t/how-to-rollover-data-stream-during-reindex/359765 "2024-05-19T15:42:15Z")
I have an index with size 5gb. I would like to reindex it to a data stream with rollover policy of max size 1gb and 7 days. This is my policy PUT \_ilm/policy/jacob\_policy { "policy": { "phases": { "hot": {… [Next page](https://discuss.elastic.co/tag/datastreams/85.md?match_all_tags=true&page=1&tags%5B%5D=datastreams)