# Help datastreams

**URL:** <https://discuss.elastic.co/t/help-datastreams/363807>\
**Category:** Elastic Search\
**Created:** [July 25, 2024, 7:37pm UTC](https://discuss.elastic.co/t/help-datastreams/363807 "2024-07-25T19:37:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![juancamiloll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juancamiloll/32/110326_2.png) [@juancamiloll](https://discuss.elastic.co/u/juancamiloll)\
**Post date:** [July 25, 2024, 7:37pm UTC](https://discuss.elastic.co/t/help-datastreams/363807/1 "2024-07-25T19:37:54Z")

</div>

The first thing I learned was how to ingest logs to indexes, and I was asked to perform this process but already using datastreams and I set myself the task of doing it.

I have already created the ILM (Index Lifecycle Management) and the index template.

I have named the policy “ **tmes\_policy** ” and the template “ **tmes\_template** ”.

This is my configuration in the .conf file

The problem I have is that the datastream is created without any problem and data is displayed from kibana but it is applying a policy called “ **logs** ” through an index template called “ **logs** ” and I don't know how to do to allow me to apply a different policy from the index template.

```auto
output {
  if [type] == "tmes" {
    elasticsearch {
      id => "xxxx_output_tmes"
      hosts => ["https://1.1.1.1:9200"]
      data_stream => true
      data_stream_type => "logs"
      data_stream_dataset => "tmes"
      data_stream_namespace => "default"
      user => "elastic"
      password => "elastic"
      ssl_enabled => true
      ssl_certificate_authorities => "/etc/logstash/certs/certificado-ca.crt"
    }
  }
}

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/3/43496eee52d7003b6ea064ac89b294f266bd9042.png)

---

<div class="post-metadata">

**Author:** ![Keith\_Massey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keith_massey/32/83666_2.png) [@Keith\_Massey](https://discuss.elastic.co/u/Keith_Massey)\
**Post date:** [July 26, 2024, 2:51pm UTC](https://discuss.elastic.co/t/help-datastreams/363807/2 "2024-07-26T14:51:41Z")

</div>

It looks like you're matching one of the built-in templates. See the "Avoid index pattern collisions" section of [Index templates | Elasticsearch Guide [8.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html). You can either use a different index pattern, or use a higher priority on your template.

---

<div class="post-metadata">

**Author:** ![juancamiloll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juancamiloll/32/110326_2.png) [@juancamiloll](https://discuss.elastic.co/u/juancamiloll)\
**Post date:** [July 26, 2024, 5:08pm UTC](https://discuss.elastic.co/t/help-datastreams/363807/3 "2024-07-26T17:08:35Z")

</div>

I think the issue goes this way, I don't know what @leandrojmp thinks.

> [@Cannot create datastream under new index template](https://discuss.elastic.co/t/cannot-create-datastream-under-new-index-template/347126):
>
> Hi I am using logstash running Kubernetes under ECK. I am ingesting both logs (filebeat) and metrics (metricbeat). Now i want ingest data from heartbeat. I get the following error: [2023-11-14T13:22:23,598][INFO][logstash.outputs.elasticsearch][main][80b6fcbcae1c789247b66f08ff48b2bde70d63d576edb68ec43e28cae9649bcd] Retrying failed action {:status=\>403, :action=\>["create", {:\_id=\>nil, :\_index=\>"testtype-http-straffe.skifte", :routing=\>nil}, {"event"=\>{"dataset"=\>"http"}, "headers"=\>{"http\_acce…

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 27, 2024, 4:48am UTC](https://discuss.elastic.co/t/help-datastreams/363807/4 "2024-07-27T04:48:23Z")

</div>

I would avoid any data stream naming scheme that could have any collision with any of the ones that elastic uses, which are `logs-*`, `metrics-*`, `synthetics-*` and `traces-*`.

You could for example use just `tmes` as your datastream name.

But this leads to another issue with is the fact that logstash does not support custom data stream names, to work with that you need to follow the workaround mentioned in the post you linked.

---

<div class="post-metadata">

**Author:** ![juancamiloll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juancamiloll/32/110326_2.png) [@juancamiloll](https://discuss.elastic.co/u/juancamiloll)\
**Post date:** [July 29, 2024, 2:13pm UTC](https://discuss.elastic.co/t/help-datastreams/363807/5 "2024-07-29T14:13:36Z")

</div>

@leandrojmp thanks for your answer, according to what little I understand the solution you propose would not be using datastream since clearly with your suggestion we would be working with normal indexes or am I wrong?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [July 29, 2024, 2:51pm UTC](https://discuss.elastic.co/t/help-datastreams/363807/6 "2024-07-29T14:51:06Z")

</div>

> [@juancamiloll](#):
>
> I understand the solution you propose would not be using datastream since clearly with your suggestion we would be working with normal indexes or am I wrong?

No, you can still use datastreams, you just need to use a custom datastream name that does not collide with the ones that elastic is already using.

But since logstash **does not** support this, you need to configure the output with `data_stream` as `false`, but in your index template you configure it to be a data stream.

This is a workaround to trick logstash into sending data to custom data stream names that it does not support yet.

For example, in your `tmes_template` you would have something like this:

```auto
{
  "index_patterns": ["tmes-*"],
  "data_stream": { },
  "composed_of": ["your-mappings", "your-settings"],
  "priority": 500
}

```

This creates a template that will match anything that starts with `tmes-*`, and it will create the indices as data streams.

Then, in your logstash output you would have something like this:

```auto
output {
  elasticsearch {
      hosts => ["HOSTS"]
      index => "tmes-logs"
      action => "create"
      http_compression => true
      data_stream => false
      manage_template => false
      ilm_enabled => false
      cacert => 'ca.crt'
      user => 'USER'
      password => 'PASSWORD'
  }
}

```

This would create a data stream named `tmes-logs` because the request will match a template that is configured to create data streams.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 26, 2024, 2:51pm UTC](https://discuss.elastic.co/t/help-datastreams/363807/7 "2024-08-26T14:51:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.