Filebeat 8.3

[abaltan]

Hello,

Hope you doing well.

we are upgrading Filebeat from 7.12 to 8.3 and we are using the config file below. But for some reason filebeat is not working. Could you please check if there is anything we need to modify on our configuration file.

filebeat.inputs:
- type: log
  enabled: true
  paths:
    
    - /opt/apigee/var/log/edge-message-processor/messagelogging/*/SF-MessageLogging/*/*/*.log
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: log

setup.ilm.enabled: false

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression

name: "apigee-dev-beat"
tags: ["apigee-dev-beat"]

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "api-log"
setup.ilm.pattern: "{now/d}-000001"





output.elasticsearch:
  hosts: ["http://IP:port"]
  # ssl.verification_mode: none
  username: [username]
  password: [passwrod]
  index: "api-log-%{[agent.version]}-%{+yyyy.MM.dd}"
  pipeline: "geoip"

setup.template.name: "api-log"
setup.template.pattern: "api-log-*"
setup.template.overwrite: true
setup.template.append_fields:
  - name: apitimestamp
    type: date
#    format: "strict_date_optional_time||epoch_millis||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ssZZ"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Thanks in advance
Regards,

[stephenb (Stephen Brown)]

@abaltan

Could you please check if there is anything we need to modify on our configuration file.

You going to need to tell us exactly what is not working? then perhaps we can help.

Are no events getting ingested?

Are events getting ingest and they are wrongly parsed?

Did you run

filebeat test config

What do you mean by not working?

Have you looked at the filebeat logs? for ERRORs?

[abaltan]

Hi @stephenb,

Thanks for your reply. The process itself is not running. as shown below

[abaltan]

Hello,

it looks like there was an issue with
setup.ilm.enabled: auto, changed auto to true and it worked.

Could you please share a document to show the difference between auto and true/false.

Thanks

[stephenb (Stephen Brown)]

Pretty easy to search and find

Configure index lifecycle management | Filebeat Reference [8.3] | Elastic

setup.ilm.enabled
Enables or disables index lifecycle management on any new indices created by Filebeat. >>> Valid values are true and false.

[abaltan]

I already read that, but I am new to elastic and it is still not clear for me. anyway thx for your time.

[stephenb (Stephen Brown)]

auto is not a valid value.

true means use the settings defined in the filebeat.yml for ILM

false mean you either do not want ILM or you have set it up manually through index template, ILM policy and bootstrap index.

[stephenb (Stephen Brown)]

@abaltan

Does the above help or do you have a more specific question.

auto is no longer a valid setting it was a valid setting in 7.x ... in the Past 7.x auto was (the default) is specified on version 7.0 and later, Filebeat automatically uses index lifecycle management if the feature is enabled in Elasticsearch and has the required license; > otherwise, Filebeat creates daily indices.

true and false is much cleaner now
Either true you use the setting in the filebeat.yml or false you do not.

ILM is a big topic.. basically the way you are doing it is daily indices (rollover)... if you want to use hot / warm etc you will need to setup and ILM policy and map it into the index template .. .by default it will use the filebeat ILM policy... below I showed you how to name your own policy which you would create.

I noticed a few other things in your config above.
You had an extra setup.ilm.enabled: false at the top

Also I cleaned up the names and the template name etc.. etc.. I think this will work better for you

You need to run filebeat setup -e to load / create the template at least once.

Here is my cleaned up version with some side notes for you. This is 8.x so it filebeat uses data streams by default perhaps take a glance at that!

filebeat.inputs:
- type: log
  enabled: true
  paths:
   -  /Users/sbrown/workspace/sample-data/discuss/container-mixed/*.log
  # json.keys_under_root: true
  # json.add_error_key: true
  # json.message_key: log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 0
  index.codec: best_compression

name: "apigee-dev-beat"
tags: ["apigee-dev-beat"]

setup.ilm.enabled: true
setup.ilm.rollover_alias: "api-log"
setup.ilm.pattern: "{now/d}-000001"

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "api-log-%{[agent.version]}"   <!---- Don't add -%{+yyyy.MM.dd} its a data strean and getts added to the backing index automatically
  # ssl.verification_mode: none
  # username: [username]
  # password: [passwrod]
  # pipeline: "geoip"

setup.template.enabled: true  <!--- You should explicitly set this 
setup.template.name: "api-log-%{[agent.version]}" <!----This needs to match the index name because now it is a data stream
setup.template.pattern: "api-log-%{[agent.version]}" <!-- Better matching Hygiene because it is a data stream 
setup.template.overwrite: false <!--- Careful with this if set to true it will overwrite every time.
setup.ilm.policy_name: api-policy <!---- If you want to use your own policy 
setup.template.append_fields:
  - name: apitimestamp
    type: date
# #    format: "strict_date_optional_time||epoch_millis||yyyy-MM-dd'T'HH:mm:ss.SSS||yyyy-MM-dd'T'HH:mm:ssZZ"

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

Results template

Data Stream

Backing index

[abaltan]

Hi @stephenb,

Thank you so much. Yeah I have read about data streams but I still need to read more tbh.

I appreciate your support.