Change data stream name in filebeat 8.*

Elastic Stack Beats
, ,
1

We used to store filebeat data from different sources in a different index due to storage size and document category using different ILM policies.

Now we can't do that because if we use ILM, we can't change the index name, so now all documents store on the same index and with the same ILM policy. That's a problem because there are policies with 1 week and others with a year that mage huge difference when it comes to storage.

Is there a way to change de data stream name for filebeat?

1 Like
2

Hello Andrés,

You can change the ilm policy the filebeat is targeting:

With this the underling index and retention is also changed.

Hope this helps

3

Thanks @Shaoranlaos

May be I wasn't clear. We need to change index name.

The thing is that rollover_alias has been removed, so I see no way to change the index name from ILM policy.

4

In case somebody reads this,

Could be an option to set filebeat output to send events to logstash?
It seems possible to handle index name and ilm in a more flexible way form logstash than filebeat.

5

@Andres_Altamirano ,

A solution might be to add a source or app id into the filebeat configuration for the input and the use logstash to split the sources into separate indices or streams, using IF / ELSE statements in Logstash.Output.

6

Here is a filebeat example that creates and writes to datastream filebeat-8.x.x-elastic-internal, index names are like .ds-filebeat-8.1.2-elastic-internal-2022.04.04-000001

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0

setup.template.name: "filebeat-%{[beat.version]}-elastic-internal"
setup.template.pattern: "filebeat-%{[beat.version]}-elastic-internal*"

setup.ilm.enabled: true
setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-elastic-internal"
setup.ilm.policy_name: "roll_daily_delete_30"


setup.kibana:
  host: "https://...:5601"
 
output.elasticsearch:
  hosts: ["https://...:9200"]
  index: "filebeat-%{[agent.version]}-elastic-internal"
  username: "..."
  password: "..."
  ssl.certificate_authorities: ["/etc/filebeat/..."]
  ssl.certificate: "/etc/filebeat/..."
  ssl.key: "/etc/filebeat/..."

processors:
  - add_host_metadata: ~
  - add_locale: ~

logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644


monitoring.enabled: false
monitoring.cluster_uuid: ...

http.enabled: true

monitoring.elasticsearch:

setup.kibana.ssl.enabled: true
setup.kibana.ssl.certificate_authorities: ["/etc/filebeat/..."]
setup.kibana.ssl.certificate: "/etc/filebeat/..."
setup.kibana.ssl.key: "/etc/filebeat/..."
1 Like
7

Edited, my ILM policy was incorrect, I fixed it and rollover is happening.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.