Filebeat setup creates more than one write index - ILM

[JGreene (Jared Greene)]

Filebeat should not attempt to create / recreate write index if a write index exists. As indices age out with ILM policies and are deleted, subsequent runs of filebeat setup attempt to create the default index of -000001 even when a later version of the index already exists.

[elasticsearch.server][WARN] path: /%3C888888-customerbeat-000001%3E, params: {index=<888888-customerbeat-000001>}
org.elasticsearch.transport.RemoteTransportException: [02-lab-es-03][172.19.18.17:9300][indices:admin/create]
Caused by: java.lang.IllegalStateException: alias [888888-customerbeat] has more than one write index [888888-customerbeat-000001,888888-customerbeat-000016]

Steps to reproduce:

1. Create index and ilm settings with Filebeat setup command.
2. Roll the index
3. Delete the initial index (-000001)
4. Run filebeat setup command.

[ChrsMark (Chris Mark)]

Hi @JGreene! Could you share Filebeat's configuration too?

@kvch any ideas here?

[JGreene (Jared Greene)]

Sure, here it is. We have filebeat, metricbeat and a custom beat running.

Following the steps above, filebeat always tries to create the -000001 for an index if it does not exist without reference to a later version of the index already present with a write alias. If ILM is enabled and indices are deleted, any new run of filebeat setup attempts to recreate the default index and write alias, resulting in the multiple write indices condition.

######################## Filebeat Configuration ############################

#==========================  Modules configuration =============================
# Reference /etc/customerbeat/modules.d/*.yml


#========================= Filebeat global options ============================

filebeat.registry.path: /var/lib/customerbeat/registry
filebeat.registry.file_permissions: 0600
filebeat.registry.flush: 0s
filebeat.overwrite_pipelines: true
#filebeat.shutdown_timeout: 0
filebeat.config:
  inputs:
    enabled: true
    #path: inputs.d/*.yml
    #reload.enabled: true
    #reload.period: 10s
  modules:
    enabled: true
    path: /etc/customerbeat/modules.d/*.yml
    #reload.enabled: true
    #reload.period: 10s

#================================ General ======================================
name: "omitted"
#tags: ["888888"]

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
  enabled: true
  loadbalance: true
  hosts: ["https://omitted:9200", "https://omitted:9200", "https://omitted:9200"]
  #compression_level: 0
  #escape_html: false
  protocol: "https"
  username: "elastic"
  password: "changeme"
  worker: 4
  index: "888888-customerbeat"
  max_retries: 3
  bulk_max_size: 3200
  backoff.init: 1s
  backoff.max: 60s
  timeout: 90
  ssl.enabled: true
  ssl.verification_mode: full
  #ssl.supported_protocols: [TLSv1.3]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca/omittedmonca.crt"]
  ssl.certificate: "/etc/filebeat/certs/omitted.crt"
  ssl.key: "/etc/filebeat/certs/omitted.key"
  #ssl.key_passphrase: ''
  #ssl.cipher_suites: []
  #ssl.curve_types: []
  #ssl.renegotiation: never

#================================= Paths ======================================
path.home: /usr/share/filebeat/
path.config: /etc/customerbeat/
path.data: /var/lib/customerbeat/data
path.logs: /var/log/customerbeat/

#================================ Keystore ====================================
# Location of the Keystore containing the keys and their sensitive values.
keystore.path: "${path.config}/beats.keystore"

#================================ Logging ======================================
logging.level: info
#logging.selectors: [ ]
logging.to_stderr: false
logging.to_syslog: false
logging.to_eventlog: false

logging.metrics.enabled: true
logging.metrics.period: 30s
logging.to_files: true
logging.files:
  path: /var/log/customerbeat/
  name: customerbeat.log
  rotateeverybytes: 10485760 # = 10MB
  keepfiles: 2
  permissions: 0600
  interval: 24h
  rotateonstartup: False
logging.json: true


# ====================== Index Lifecycle Management (ILM) ======================
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "888888-customerbeat"
setup.ilm.pattern: "000001"
setup.ilm.policy_name: "888888-customerbeat_policy"
setup.ilm.policy_file: /etc/customerbeat/customerbeat_ilm.json
setup.template:
  name: "888888-customerbeat"
  pattern: "888888-*"
  enabled: true
setup.ilm.overwrite: true
setup.dashboards:
  enabled: true
  #index: "888888-*"
setup.kibana.space.id: "888888"

#============================== X-Pack Monitoring ===============================
monitoring.enabled: false
#monitoring.cluster_uuid:
monitoring.elasticsearch:
  hosts: ["https://omitted:9200", "https://omitted:9200", "https://omitted:9200"]
  compression_level: 1
  protocol: "https"
  username: "elastic"
  password: "changeme"
  max_retries: 3
  bulk_max_size: 50
  backoff.init: 1s
  backoff.max: 60s
  timeout: 90
  ssl.enabled: true
  ssl.verification_mode: full
  #ssl.supported_protocols: [TLSv1.3]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca/omittedmonca.crt"]
  ssl.certificate: "/etc/filebeat/certs/omitted.crt"
  ssl.key: "/etc/filebeat/certs/omitted.key"
  #ssl.key_passphrase: ''
  #ssl.cipher_suites: []
  #ssl.curve_types: []
  #ssl.renegotiation: never
  metrics.period: 1m
  state.period: 10m

# ================================= Dashboards =================================
setup.kibana:
  host: "https://omitted:5601"
  protocol: "https"
  username: "elastic"
  password: "changeme "
  path: ""
  ssl.enabled: true
  ssl.verification_mode: full
  #ssl.supported_protocols: [TLSv1.3]
  ssl.certificate_authorities: ["/etc/filebeat/certs/ca/omittedmonca.crt"]
  ssl.certificate: "/etc/filebeat/certs/omitted.crt"
  ssl.key: "/etc/filebeat/certs/omitted.key"

#================================ HTTP Endpoint ======================================
http.enabled: true
http.host: "omitted"
http.port: 5062

#============================= Process Security ================================

# Enable or disable seccomp system call filtering on Linux. Default is enabled.
seccomp.enabled: true

#================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: false

[JGreene (Jared Greene)]

Additionally, we are on 7.15.2 (Beats, Elastic, & Kibana). However, this appears to happen for previous versions as well.