# Issue with Logstash, ILM and Data stream

**URL:** <https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897>\
**Category:** Logstash\
**Tags:** ilm-index-lifecycle-management\
**Created:** [August 14, 2024, 8:41am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897 "2024-08-14T08:41:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![fio23](https://avatars.discourse-cdn.com/v4/letter/f/7ba0ec/32.png) [@fio23](https://discuss.elastic.co/u/fio23)\
**Post date:** [August 14, 2024, 8:41am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897/1 "2024-08-14T08:41:19Z")

</div>

Hello all,  
I'm currently trying to create a data stream with a corresponding ILM policy enabled, and then have Logstash to forward logs to that data stream.

See below steps I follow:

**Create Component Template**

```auto
PUT _component_template/comp_header_template
{
  "template": {
    "settings": {
      "index.lifecycle.name": "pbheaderpolicy",
      "index.lifecycle.rollover_alias": "header_stream"
    }
  }
}

```

**Create Index Template**

```auto
PUT _index_template/index_header_template
{
  "index_patterns": ["header_stream*"],
  "data_stream": { },
  "composed_of": ["comp_header_template"],
  "priority": 500
}

```

Mind that I have a customer ILM Policy created named _pbheaderpolicy_ which is included inside the component template.

**Logstash Output**

```auto
elasticsearch {
  hosts => ["https://es01:9200"]
  user => "elastic"
  password => " *****"
  ssl_enabled => true
  cacert => "/usr/share/logstash/certs/ca/ca.crt"
  data_stream => "true"
  data_stream_type => "logs"
  data_stream_dataset => "header_stream"
  data_stream_namespace => "development"
}

```

Logstash works without any errors, but the data stream is getting different index template and ilm policy, as per the below:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/2/62e36ffc346dd070e485be6791d469afdbde9201.png)

I have been trying multiple outputs on logstash, but apparently it doesn't seem to be working properly..

Any assistance or further guidance over this, would be highly appreciated!  
Thanks in advance.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 14, 2024, 12:40pm UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897/2 "2024-08-14T12:40:31Z")

</div>

Hello and welcome,

It seems that you want to create a custom data stream named `header_stream`, the issue here is that logstash does not support natively the creation of custom data streams.

Logstash only creates datastreams that follows the default name pattern, which is something like `logs-<dataset>-<namespace>`, which you can confirm since the data stream `logs-header_stream-development` was created.

There is a workaround to be able to create custom data streams, you basically set `data_stream` to `false`, put the name of the data stream rollover alias in the `index` option and set the `action` to `create`.

Something like this in your case:

```auto
output {
    elasticsearch {
        hosts => ["HOSTS"]
        index => "header_stream"
        action => "create"
        http_compression => true
        data_stream => false
        manage_template => false
        ilm_enabled => false
        cacert => 'ca.crt'
        user => 'USER'
        password => 'PASSWORD'
    }
}

```

The options `manage_template` and `ilm_enabled` are also set to `false` because the ILM policy and the template already exists in Elasticsearch.

I created an issue about this last year, but still no updates: [Allow the creation of custom data streams · Issue #1152 · logstash-plugins/logstash-output-elasticsearch · GitHub](https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/1152)

---

<div class="post-metadata">

**Author:** ![fio23](https://avatars.discourse-cdn.com/v4/letter/f/7ba0ec/32.png) [@fio23](https://discuss.elastic.co/u/fio23)\
**Post date:** [August 14, 2024, 1:41pm UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897/3 "2024-08-14T13:41:09Z")

</div>

Thanks a lot my friend, this works fine!

---

<div class="post-metadata">

**Author:** ![Barrow\_Kwan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/barrow_kwan/32/46413_2.png) [@Barrow\_Kwan](https://discuss.elastic.co/u/Barrow_Kwan)\
**Post date:** [December 10, 2024, 12:24am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897/4 "2024-12-10T00:24:02Z")

</div>

I am having the same issue when I configure logstash to create the datastream, it default to use "log" template and "log" ILM.

with the suggestion you provided, I will need to create the datasream manually right? I still not sure how it will use the data\_stream if data\_stream is set to true in logstash's ES output?

thanks

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 10, 2024, 4:30am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897/5 "2024-12-10T04:30:54Z")

</div>

> [@Barrow\_Kwan](#):
>
> with the suggestion you provided, I will need to create the datasream manually right?

No, you need to create a data stream template first, like the example in the first post.

Something like this:

```auto
PUT _index_template/your-index-template
{
  "index_patterns": ["your-data-stream-pattern"],
  "data_stream": { },
  "composed_of": ["comp_header_template"],
  "priority": 500
}

```

Then you use an output like the one I suggested, with `data_stream => false`.

> [@Barrow\_Kwan](#):
>
> I still not sure how it will use the data\_stream if data\_stream is set to true in logstash's ES output?

If you set `data_stream` as `true` in the Logstash Elasticsearch output, your datastream name will need to be something like `logs-dataset-namespace` and it will use the `logs` template and `log` ILM.

As mentioned in my previous answer logstash does not support custom data streams, the only way to write to custom data streams with different name patterns is with the workaround shared.

---

<div class="post-metadata">

**Author:** ![Barrow\_Kwan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/barrow_kwan/32/46413_2.png) [@Barrow\_Kwan](https://discuss.elastic.co/u/Barrow_Kwan)\
**Post date:** [December 19, 2024, 7:28am UTC](https://discuss.elastic.co/t/issue-with-logstash-ilm-and-data-stream/364897/6 "2024-12-19T07:28:34Z")

</div>

@leandrojmp thanks a lot. I was able to configure the custom datastream now.