Filebeat ILM policy question

[Mark_Visser]

Hello,

For my filebeat rollover to work it expects an alias to be set on the index. In the past you were able to pass along a rollover_alias in the filebeat config but that is not possible anymore.

I found you can manually set an alias on in index using PUT /filebeat-8.4.2/_alias/filebeat, but is it possible to include this in the index template so this manual step can be skipped?

Thanks for your time.

[stephenb (Stephen Brown)]

Hi @Mark_Visser Welcome to the community!

Yes, the docs don't do a great job talking about data streams at this point.

But I think there's a pretty good example here, You still use all the existing settings. It just writes to a data stream.

EDIT : I put a Super Simple Example Below...

[Mark_Visser]

Thanks for the reply Stephen,

When following the post you linked, should the newly created index show up under Data Streams?

I find it under the indices tab as usual, thanks. I'll test if rollovers work correctly now.

[stephenb (Stephen Brown)]

Hmmm let me look I am actually reproducing over here... I assume you want a data stream correct? or do you want an index?

[Mark_Visser]

With the previous post you linked it's supposed to be using a data stream in elastic. But following that post myself doesn't seem to create it as a data stream.

This is currently my config, please note that indenting might look slightly different cause I run it on kubernetes;

      filebeat.yml: |
        filebeat.inputs:
        - type: container
          paths:
            - /var/log/containers/*.log
          processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
              - logs_path:
                  logs_path: "/var/log/containers/"

        output.elasticsearch:
          hosts: ["https://myelasticsearchhost:443"]
          username: ${FILEBEAT_USER}
          password: ${FILEBEAT_PASSWORD}
          index: "filebeat-%{[agent.version]}-elastic-internal"

          ssl.verification_mode: none

        setup.template.name: "filebeat-%{[beat.version]}-elastic-internal"
        setup.template.pattern: "filebeat-%{[beat.version]}-elastic-internal*"

        setup.ilm.enabled: true
        setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-elastic-internal"
        setup.ilm.policy_name: "roll_daily_delete_30"

[stephenb (Stephen Brown)]

Yes I did a search before I verified

I always test on a local host before I try K8s... less variables.

Here is my complete filebeat.yml which produces a data stream

You will need to create an Template and ILM policies if you want

With data streams the data stream is basically the rollover alias everything is "hidden" behind it

This is the minimum and it works

filebeat.inputs:
- type: log
  enabled: true
  paths:
   -  /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log

setup.template.name: "myindex-%{[beat.version]}"
setup.template.pattern: "myindex-%{[beat.version]}*"

setup.kibana:
 
output.elasticsearch:
  hosts: ["http://localhost:9200"]
  index: "myindex-%{[agent.version]}"

filebeat -e

GET _data_stream/

{
  "data_streams": [
    {
      "name": "myindex-8.4.1",
      "timestamp_field": {
        "name": "@timestamp"
      },
      "indices": [
        {
          "index_name": ".ds-myindex-8.4.1-2022.10.07-000001",
          "index_uuid": "-TCtT0j1SVeh6RG1risylQ"
        }
      ],
      "generation": 1,
      "status": "GREEN",
      "template": "myindex-8.4.1",
      "ilm_policy": "filebeat",
      "hidden": false,
      "system": false,
      "allow_custom_routing": false,
      "replicated": false
    }
  ]
}

GET _cat/indices/myindex-*?v

health status index                               uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .ds-myindex-8.4.1-2022.10.07-000001 -TCtT0j1SVeh6RG1risylQ   1   0          5            0     10.5kb         10.5kb

[Mark_Visser]

Hi Stephen,

Good news

Guess I'll have to read up a bit on the ILM / templating changes so I can add those too.
Thanks for the assistance and have a good weekend.

[stephenb (Stephen Brown)]

Hi @Mark_Visser Good!

Kinda funny thing if you just wanted filebeat-8.4.1 you did not need to do anything... that is the default... quite literally you did not need to set anything that is the default behavior.

Often when people get started I highly advise the to start with the defaults ... get things working then if you want to rename indices / data streams etc then do it.

I thought you wanted to rename the data stream that is what I showed you..

If you are just getting started don't do that and let all the goodness of the defaults take place

[Mark_Visser]

I'll mark this as the solution. Thanks for the help mate.