# Split a data stream into separate Indexes - to allow different ILM policies **URL:** \ **Category:** Elastic Agent\ **Tags:** integrations\ **Created:** [August 20, 2024, 5:54pm UTC](https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223 "2024-08-20T17:54:11Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mgordon](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mgordon](https://discuss.elastic.co/u/mgordon)\ **Post date:** [August 20, 2024, 5:54pm UTC](https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223/1 "2024-08-20T17:54:11Z")
I'm using the Custom UDP Logs integration and need to split the data stream into two separate indexes (data streams), so I can enable different ILM policies on the two different types of data coming in. Is that possible, or do I need to use two different integrations coming from the same source? Suggestions? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [August 20, 2024, 6:32pm UTC](https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223/2 "2024-08-20T18:32:58Z")
Hi @mgordon Have you looked at the `reroute` processor... with a condition... made for just such a use case > **[Reroute processor | Elasticsearch Guide \[8.15\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/reroute-processor.html)** ---
**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\ **Post date:** [August 20, 2024, 6:33pm UTC](https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223/3 "2024-08-20T18:33:12Z")
First of all, the Custom UDP Logs and TCP Logs integrations do not do their best job into helping the end user setup the proper index template structure. In the Fleet Integration we offer to customize the `Dataset name`, but the Integration will not do anything for you automatically. It will install, by default, the Index Template `logs-udp.generic` regardless of the `Dataset name` we've chosen. Note the Custom Logs integration is slightly different. If you set the `Dataset name` to `my.dataset`, it will automatically create for you an Index Template which will match `logs-my.dataset` so that you have a good starting point. Now, back to the **Custom UDP Log integration**. 1. Is the data the same structure/type, but it's just related to different environments? 2. Can the data be received on different ports or hosts? If the data is the same and you cannot receive it from different ports/hosts, then I would go with 1 integration and then route the events using an ingest pipeline. This is based on 8.14-ish... # Approach A Create a Custom UDP Integration with the `Dataset name` set to `my.dataset`. I would recommend having an Index Template for a `-` index (e.g. `logs-my.dataset` Index template) completely decoupled with the `udp.generic` one. To create the `logs-my.dataset` Index Template, you can clone the `logs-udp.generic-*` (and all the associated component templates) - Index Template `logs-my.dataset` (cloned from `logs-udp.generic`, with index pattern `logs-my.dataset-*`) - Component Templates: - `logs@mappings` - `logs@settings` - `logs-my.dataset@package`, cloned from `logs-udp.generic@package`, but: - replace the `default_pipeline` with `logs-my.dataset` - then create an empty ingest pipeline - define the type of the fields you will expect in the mappings section - `logs-my.dataset@custom` - `ecs@mappings` - `.fleet_globals-1` - `.fleet_agent_id_verification-1` Once you have this, you can clone the Index Template `logs-my.dataset` into: - Index Template `logs-my.dataset-namespace1` with index pattern `logs-my.dataset-namespace1` and custom ILM Policy `n1` - Index Template `logs-my.dataset-namespace2` with index pattern `logs-my.dataset-namespace2` and custom ILM Policy `n2` How do you route the events to different datastreams? Using the `logs-my.dataset` ingest pipeline with the `reroute` processor. You can [reroute](https://www.elastic.co/guide/en/elasticsearch/reference/current/reroute-processor.html) events to a different `namespace` using a conditional. This should somewhat ensure this will not break on future changes as here we're literally decoupling the behavior from the `Custom UDP Logs` assets. # Approach B If instead you are ok with keeping `udp.generic` as `Dataset name`, then you can: - Clone the Index Template `logs-udp.generic` into 2 Index Templates - Index Template `logs-udp.generic-namespace1` with index pattern `logs-udp.generic-namespace1` and custom ILM Policy `n1` - Index Template `logs-udp.generic-namespace2` with index pattern `logs-udp.generic-namespace2` and custom ILM Policy `n2` - Define an ingest pipeline `logs-udp.generic@custom` with the `reroute` processor to route to `namespace1` or `namespace2` * * * Approach A allows you to have your own "assets". - If you have to modify the Index Template, you have to do it 3 times. Approach B makes you dependent on all the assets of `upd.generic` which can be an advantage/disadvantage. - If you have to modify the Index Template, you have to do it 3 times. - If the Index Template structure get changed by Fleet (enhancements, etc...), you might need to re-clone to align with the new Index Template of the Custom UDP ---
**Author:** ![mgordon](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mgordon](https://discuss.elastic.co/u/mgordon)\ **Post date:** [August 20, 2024, 7:21pm UTC](https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223/4 "2024-08-20T19:21:42Z")
I hadn't looked at that Reroute processor - it does exactly what I need! I tried to update the dataset or namespace manually in the pipeline, bit it failed becuase the value wasn't allowed in the constant-keyword datatype. Thank you! ---
**Author:** ![mgordon](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mgordon](https://discuss.elastic.co/u/mgordon)\ **Post date:** [August 20, 2024, 7:22pm UTC](https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223/5 "2024-08-20T19:22:43Z")
Thank you - I'm essentially using Approach B, but hadn't seen the Reroute processor, which was the missing piece. Thank you!