Rollover not working, Filebeat default index does not have an alias

Elastic Stack Elasticsearch
1

Hello,

I would like to use rollover to delete all logs, however, I always get an error message. It does not work. I can use only if turn of rollover. I do not modify anything on indexes, I use default Indexes after installing.

Untitled-1
Untitled-11920×866 63 KB

I used the following commands for installing:

  • sudo filebeat modules enable system apache mysql
  • sudo filebeat setup --pipelines --modules system apache mysql
  • sudo filebeat setup --pipelines --modules system -M "system.syslog.enabled=true" -M "system.auth.enabled=true"
  • sudo filebeat setup --pipelines -e

Is it possible to configure Filebeat, Metricbeat, etc to automaticlaly create an alias whenever the index rolls over a to a new one due to ILM lifecycle policy?

How can I fix this issue? I have already tried to delete DataStream, after then delete Indexes, Index Templates but it does not help me.

2

How is your Filebeat Elasticsearch output configured?

3 
filebeat.inputs:


- type: filestream

  id: my-filestream-id-1

  enabled: true

  paths:
    - /var/log/*.log


filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: true

  reload.period: 10s


setup.template.settings:
  index.number_of_shards: 1

tags: ["ultron", "filebeat"]

fields:
  env: prod
  part: filebeat

setup.kibana:

  host: "https://kibana.xxxxx.net:5601"
  ssl.enabled: true
  ssl.certificate_authorities: "/etc/filebeat/certs/ca.crt"
  ssl.certificate: "/etc/filebeat/certs/filebeat.crt"
  ssl.key: "/etc/filebeat/certs/filebeat.key"

output.elasticsearch:
  hosts: ["elastic.xxxxxxx.net:9200"]

  ssl.certificate_authorities: "/etc/filebeat/certs/ca.crt"
  ssl.certificate: "/etc/filebeat/certs/filebeat.crt"
  ssl.key: "/etc/filebeat/certs/filebeat.key"

  protocol: "https"

  username: "elastic"
  password: "xxxxxxxxxxxxx"

processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
4

Hi,

you can read this:

Regards

5

Added following command at the end of filebeat.yml:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

setup:
  ilm:
    check_exists: false
    enabled: true
    overwrite: true
    pattern: "%{now/d}-000001"
    rollover_alias: "filebeat-%{[agent.version]}"
    policy_name: filebeat

Then I deleted templates, datastream and ran that command:
cd /usr/share/filebeat/bin && ./filebeat setup -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.data /var/lib/filebeat

Unfortunately, I always get this error message.
Policy filebeat is configured for rollover, but index .ds-filebeat-8.11.4-2024.01.14-000001 does not have an alias, which is required for rollover.

What wrong? What should I do?

6

Hi @whanklee

We need to back way up..

What version are you on?

I think we are making this too complicated.

Plus you are using data streams and trying to treat them configure them like indices.

You do not rollover data streams by pointing to the .ds-... index

Tell me what version you are on and top level what you are trying to accomplish.

In short you should take out all the non default settings out of the filebeat.yml

The we will work with the data stream and the ILM policy

7

Version filebeat-8.11.4.

I want to delete old logs in index. Filebeat uses default index with name .ds-filebeat-8.11.4-2024.01.14-000001. I want to use rollover.

1
12828×634 183 KB

So I override default settings in policy filebeat. I want to start writing new index when container size is more than 10 MB for example.

2
21920×1051 83.4 KB

3
31920×1081 78.4 KB

8

Okay...

You are mostly right, but you're trying to treat the data stream which you should read about like an normal index. You should read about data streams

Just to warn you when you set ILM to settings like megabytes or minutes for rollover, it's not going to operate the way you think it is because ILM is meant to be used at scale on the gigabytes an hours. I'm just letting you know that you can test but do not expect it to exactly roll over on 10 megabytes. ILM Is a background process.

Third everything else in your filebeat.yml that you added about ILM etc. That is not useful.

4th always run the full setup command Not the setup command for individual components like index management

filebeat setup -e

Set your new ILM policy to what you want and again, when you set these tiny settings it's not going to work exactly how you want it to work.... And it does work at scale.

Now to roll over a datastream you do not try to roll over the actual index You roll over the data stream

POST filebeat-8.11.4/_rollover

Then run get

GET .ds-filebeat-8.11.4-2024.01.14-000001/_ilm/explain

Let's see how far you get.

I appreciate you jumping in and just trying everything, but it's probably worthwhile understanding data streams and ILM. I read those sections of the documentation

9

Thank you, it works.

1
12816×882 268 KB

{
  "acknowledged": true,
  "shards_acknowledged": true,
  "old_index": ".ds-filebeat-8.11.4-2024.01.14-000004",
  "new_index": ".ds-filebeat-8.11.4-2024.01.14-000005",
  "rolled_over": true,
  "dry_run": false,
  "conditions": {}
}
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.