Filebeat Elasticsearch output data stream?

Elastic Search
1

Hi folks,

do I understand right? For filebeat there is still no option to define data stream in the elasticsearch output like so:

  output.elasticsearch:
    hosts: ["https://elastic-cloud-endpoint.com:9243"]
    username: ${USER}
    password: ${PASS}
    data_stream_type: logs
    data_stream_dataset: my.app
    data_stream_namespace: prod
    workers: 2

I a still left with an option to define only an index and creating template and ilm policy beforehand manually or via filebeat (even if I do not want to, because I want to use the default templates and policies?)

Thanksk

2

You will just set the index to the data stream name: logs-my.app-prod

Whether a data stream is created just depends on whether or not there's an index template which matches the index name and that it is defined to create a data stream

If you set the index to mydata-prod, it will not match any index templates and will result in a regular index being created.

By default anything that starts with logs-* should result in a datastream

2 Likes
3

Example: Can be set per input

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log
  fields_under_root: true
  fields:
    data_stream.type: logs
    data_stream.dataset: my.app
    data_stream.namespace: prod
    event.dataset: my.app

setup.ilm.enabled: false
setup.template.enabled: false
# setup.template.settings:
#   index.number_of_shards: 1

setup.kibana:

output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "%{[data_stream.type]}-%{[data_stream.dataset]}-%{[data_stream.namespace]}"
1 Like
4

Thanks that is what I was looking for. This info would be really helpful on the filebeat input documentation actually. There is only example with index.

I do understand right using the naming vention logs-*-* I do not need to create a template for that right? It should take the default/managed logs template.

5

That is correct