Does the above help or do you have a more specific question.
auto
is no longer a valid setting it was a valid setting in 7.x ... in the Past 7.xauto
was (the default) is specified on version 7.0 and later, Filebeat automatically uses index lifecycle management if the feature is enabled in Elasticsearch and has the required license; > otherwise, Filebeat creates daily indices.
true
andfalse
is much cleaner now
Eithertrue
you use the setting in the filebeat.yml orfalse
you do not.
ILM is a big topic.. basically the way you are doing it is daily indices (rollover)... if you want to use hot / warm etc you will need to setup and ILM policy and map it into the index template .. .by default it will use the filebeat ILM policy... below I showed you how to name your own policy which you would create.
I noticed a few other things in your config above.
You had an extra setup.ilm.enabled: false
at the top
Also I cleaned up the names and the template name etc.. etc.. I think this will work better for you
You need to run filebeat setup -e
to load / create the template at least once.
Here is my cleaned up version with some side notes for you. This is 8.x so it filebeat uses data streams by default perhaps take a glance at that!
filebeat.inputs:
- type: log
enabled: true
paths:
- /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log
# json.keys_under_root: true
# json.add_error_key: true
# json.message_key: log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 0
index.codec: best_compression
name: "apigee-dev-beat"
tags: ["apigee-dev-beat"]
setup.ilm.enabled: true
setup.ilm.rollover_alias: "api-log"
setup.ilm.pattern: "{now/d}-000001"
output.elasticsearch:
hosts: ["localhost:9200"]
index: "api-log-%{[agent.version]}" <!---- Don't add -%{+yyyy.MM.dd} its a data strean and getts added to the backing index automatically
# ssl.verification_mode: none
# username: [username]
# password: [passwrod]
# pipeline: "geoip"
setup.template.enabled: true <!--- You should explicitly set this
setup.template.name: "api-log-%{[agent.version]}" <!----This needs to match the index name because now it is a data stream
setup.template.pattern: "api-log-%{[agent.version]}" <!-- Better matching Hygiene because it is a data stream
setup.template.overwrite: false <!--- Careful with this if set to true it will overwrite every time.
setup.ilm.policy_name: api-policy <!---- If you want to use your own policy
setup.template.append_fields:
- name: apitimestamp
type: date
# # format: "strict_date_optional_time||epoch_millis||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ssZZ"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
Results template
Data Stream
Backing index