# Filebeat Elasticsearch output data stream?

**URL:** <https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562>\
**Category:** Elastic Search\
**Created:** [December 5, 2024, 7:33pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562 "2024-12-05T19:33:55Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kosmodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosmodrom/32/134481_2.png) [@Kosmodrom](https://discuss.elastic.co/u/Kosmodrom)\
**Post date:** [December 5, 2024, 7:33pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/1 "2024-12-05T19:33:55Z")

</div>

Hi folks,

do I understand right? For filebeat there is still no option to define data stream in the elasticsearch output like so:

```
  output.elasticsearch:
    hosts: ["https://elastic-cloud-endpoint.com:9243"]
    username: ${USER}
    password: ${PASS}
    data_stream_type: logs
    data_stream_dataset: my.app
    data_stream_namespace: prod
    workers: 2

```

I a still left with an option to define only an index and creating template and ilm policy beforehand manually or via filebeat (even if I do not want to, because I want to use the default templates and policies?)

Thanksk

---

<div class="post-metadata">

**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\
**Post date:** [December 6, 2024, 3:12am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/2 "2024-12-06T03:12:48Z")

</div>

You will just set the index to the data stream name: `logs-my.app-prod`

Whether a data stream is created just depends on whether or not there's an index template which matches the index name and that it is defined to create a data stream

If you set the index to `mydata-prod`, it will not match any index templates and will result in a regular index being created.

By default anything that starts with `logs-*` should result in a datastream

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 6, 2024, 3:54am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/3 "2024-12-06T03:54:11Z")

</div>

Example: Can be set per input

```auto
filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log
  fields_under_root: true
  fields:
    data_stream.type: logs
    data_stream.dataset: my.app
    data_stream.namespace: prod
    event.dataset: my.app

setup.ilm.enabled: false
setup.template.enabled: false
# setup.template.settings:
# index.number_of_shards: 1

setup.kibana:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"

```

---

<div class="post-metadata">

**Author:** ![Kosmodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosmodrom/32/134481_2.png) [@Kosmodrom](https://discuss.elastic.co/u/Kosmodrom)\
**Post date:** [December 6, 2024, 8:17am UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/4 "2024-12-06T08:17:33Z")

</div>

Thanks that is what I was looking for. This info would be really helpful on the filebeat input documentation actually. There is only example with index.

I do understand right using the naming vention `logs-*-*` I do not need to create a template for that right? It should take the default/managed `logs` template.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [December 6, 2024, 2:54pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/5 "2024-12-06T14:54:18Z")

</div>

> [@Kosmodrom](#):
>
> I do not need to create a template for that right? It should take the default/managed `logs` template.

That is correct

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 3, 2025, 2:54pm UTC](https://discuss.elastic.co/t/filebeat-elasticsearch-output-data-stream/371562/6 "2025-01-03T14:54:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.