Filebeat 8.3

stephenb · September 19, 2022, 12:54am

Does the above help or do you have a more specific question.

auto is no longer a valid setting it was a valid setting in 7.x ... in the Past 7.x auto was (the default) is specified on version 7.0 and later, Filebeat automatically uses index lifecycle management if the feature is enabled in Elasticsearch and has the required license; > otherwise, Filebeat creates daily indices.

true and false is much cleaner now
Either true you use the setting in the filebeat.yml or false you do not.

ILM is a big topic.. basically the way you are doing it is daily indices (rollover)... if you want to use hot / warm etc you will need to setup and ILM policy and map it into the index template .. .by default it will use the filebeat ILM policy... below I showed you how to name your own policy which you would create.

I noticed a few other things in your config above.
You had an extra setup.ilm.enabled: false at the top

Also I cleaned up the names and the template name etc.. etc.. I think this will work better for you

You need to run filebeat setup -e to load / create the template at least once.

Here is my cleaned up version with some side notes for you. This is 8.x so it filebeat uses data streams by default perhaps take a glance at that!

filebeat.inputs:
- type: log
  enabled: true
  paths:
   -  /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log
  # json.keys_under_root: true
  # json.add_error_key: true
  # json.message_key: log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression

name: "apigee-dev-beat"
tags: ["apigee-dev-beat"]

setup.ilm.enabled: true
setup.ilm.rollover_alias: "api-log"
setup.ilm.pattern: "{now/d}-000001"

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "api-log-%{[agent.version]}"   <!---- Don't add -%{+yyyy.MM.dd} its a data strean and getts added to the backing index automatically
  # ssl.verification_mode: none
  # username: [username]
  # password: [passwrod]
  # pipeline: "geoip"

setup.template.enabled: true  <!--- You should explicitly set this 
setup.template.name: "api-log-%{[agent.version]}" <!----This needs to match the index name because now it is a data stream
setup.template.pattern: "api-log-%{[agent.version]}" <!-- Better matching Hygiene because it is a data stream 
setup.template.overwrite: false <!--- Careful with this if set to true it will overwrite every time.
setup.ilm.policy_name: api-policy <!---- If you want to use your own policy 
setup.template.append_fields:
  - name: apitimestamp
    type: date
# #    format: "strict_date_optional_time||epoch_millis||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ssZZ"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Results template

Data Stream

Backing index