Upgrade Filebeat from 6.7 to 7.0

[PWIT]

I have upgraded my filebeat version from 6.7 to 7.0, here is my filebeat.yml

#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /usr/local/app/trade-service/logs/*.json.log
  exclude_files: ['\.gz$']
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: message
  fields: {service-instance: "01"}
  fields_under_root: true

- type: log
  enable: true
  paths:
    - /usr/local/app/trade-client/logs/*.json.log
  exclude_files: ['\.gz$']
  json.keys_under_root: true
  json.add_error_key: true
  json.message_key: message
#============================= Filebeat modules ===============================
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 60s
#==================== Template ==========================
setup.template.enabled: true
setup.template.name: "app-trade-%{[agent.version]}"
setup.template.pattern: "app-trade-%{[agent.version]}-*"
setup.template.fields: "${path.config}/fields.yml"
setup.template.overwrite: true
setup.template.settings:
  index.number_of_shards: 3
  index.number_of_replicas: 0
  #index.codec: best_compression
  #_source.enabled: false
#================================ General =====================================
name: "app-trade-shipper"
fields:
  firm: "XEBT"
  env: "dev"
fields_under_root: true
#tags: ["service-X", "web-tier"]

#============================== Dashboards =====================================
#setup.dashboards.enabled: false
#setup.dashboards.url:

#============================== Kibana =====================================
setup.kibana:
  host: "192.168.7.25:5601"
  #space.id:
#============================= Elastic Cloud ==================================
#cloud.id:
#cloud.auth:

#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  enable: true
  hosts: ["192.168.7.25:9200"]
  index: "app-trade-%{[agent.version]}-%{+yyyy.MM.dd}"
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  #hosts: ["localhost:5044"]
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  #ssl.certificate: "/etc/pki/client/cert.pem"
  #ssl.key: "/etc/pki/client/cert.key"
#================================ Processors =====================================
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
#================================ Logging =====================================
logging.level: info
#logging.selectors: ["*"]
logging.to_syslog: false
logging.to_eventlog: false
logging.metrics.enabled: true
logging.metrics.period: 60s
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  rotateeverybytes: 10485760
  keepfiles: 7
  permissions: 0600

logging.json: false

#============================== Xpack Monitoring ===============================
# reporting is disabled by default.
#xpack.monitoring.enabled: false
#xpack.monitoring.elasticsearch:

#================================= Migration ==================================
#migration.6_to_7.enabled: true

After adjusting the config-file "filebeat.yml" in directory "/etc/filebeat" the following problems occur.

• The index "app-trade-%{[agent.version]}-%{+yyyy.MM.dd}" is not created but always the index "filebeat-7.0.0", but this worked fine in version 6.7.

• Logging is always done into syslog

logs from syslog:

filebeat[3310]: 2019-04-12T12:25:37.734+0200#011INFO#011[index-management]#011idxmgmt/std.go:361#011Set setup.template.name to '{filebeat-7.0.0 {now/d}-000001}' as ILM is enabled.
filebeat[3310]: 2019-04-12T12:25:37.734+0200#011INFO#011[index-management]#011idxmgmt/std.go:366#011Set setup.template.pattern to 'filebeat-7.0.0-*' as ILM is enabled.
filebeat[3310]: 2019-04-12T12:25:37.734+0200#011INFO#011[index-management]#011idxmgmt/std.go:400#011Set settings.index.lifecycle.rollover_alias in template to {filebeat-7.0.0 {now/d}-000001} as ILM is enabled.
filebeat[3310]: 2019-04-12T12:25:37.734+0200#011INFO#011[index-management]#011idxmgmt/std.go:404#011Set settings.index.lifecycle.name in template to {filebeat-7.0.0 map[policy:{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}]} as ILM is enabled.
filebeat[3310]: 2019-04-12T12:25:37.736+0200#011INFO#011template/load.go:82#011Loading template for Elasticsearch version: 7.0.0
filebeat[3310]: 2019-04-12T12:25:37.737+0200#011INFO#011template/load.go:84#011Existing template will be overwritten, as overwrite is enabled.
filebeat[3310]: 2019-04-12T12:25:37.913+0200#011INFO#011template/load.go:145#011Elasticsearch template with name 'filebeat-7.0.0' loaded
filebeat[3310]: 2019-04-12T12:25:37.913+0200#011INFO#011[index-management]#011idxmgmt/std.go:272#011Loaded index template.

Thank you for your help and best regards, Peter.

[kierandelaney]

You’ve got ILM turned on (see your log messages). You’ll need to specifically disable it in 7.0

[suikast42]

I am using the folowing output config, But still the same behaviour. Only filebeat-xx is created:

  output.elasticsearch:
hosts: ["my.cloud:9200"]
index: "es-logstream-%{+yyyy.MM.dd}"
pipeline: "ingest-logs"
worker: 3
  setup.template:
name: es-logstream-mapping
pattern: es-logstream*
enabled: true
overwrite: true
ilm.enabled: false

[PWIT]

it's works with (as in filebeat.refernces.yml in a standalone section not under output.elasticsearc)

#============================== Setup ILM =====================================
setup.ilm.enabled: false

best regards Peter

[suikast42]

Ok. That works thanks.

That was ltlle confisuing to have two similar keys like

setup.template.ilm.enabled: false and
setup.ilm.enabled: false

[kierandelaney]

Yeah it does look a bit confusing. We had a very similar issue!