JGreene
(Jared Greene)
December 1, 2021, 2:16pm
1
Filebeat should not attempt to create / recreate write index if a write index exists. As indices age out with ILM policies and are deleted, subsequent runs of filebeat setup attempt to create the default index of -000001
even when a later version of the index already exists.
[elasticsearch.server][WARN] path: /%3C888888-customerbeat-000001%3E, params: {index=<888888-customerbeat-000001>}
org.elasticsearch.transport.RemoteTransportException: [02-lab-es-03][172.19.18.17:9300][indices:admin/create]
Caused by: java.lang.IllegalStateException: alias [888888-customerbeat] has more than one write index [888888-customerbeat-000001,888888-customerbeat-000016]
Steps to reproduce:
Create index and ilm settings with Filebeat setup command.
Roll the index
Delete the initial index (-000001)
Run filebeat setup command.
ChrsMark
(Chris Mark)
December 2, 2021, 8:14am
2
Hi @JGreene ! Could you share Filebeat's configuration too?
@kvch any ideas here?
JGreene
(Jared Greene)
December 2, 2021, 12:53pm
3
Sure, here it is. We have filebeat, metricbeat and a custom beat running.
Following the steps above, filebeat always tries to create the -000001
for an index if it does not exist without reference to a later version of the index already present with a write alias. If ILM is enabled and indices are deleted, any new run of filebeat setup
attempts to recreate the default index and write alias, resulting in the multiple write indices condition.
######################## Filebeat Configuration ############################
#========================== Modules configuration =============================
# Reference /etc/customerbeat/modules.d/*.yml
#========================= Filebeat global options ============================
filebeat.registry.path: /var/lib/customerbeat/registry
filebeat.registry.file_permissions: 0600
filebeat.registry.flush: 0s
filebeat.overwrite_pipelines: true
#filebeat.shutdown_timeout: 0
filebeat.config:
inputs:
enabled: true
#path: inputs.d/*.yml
#reload.enabled: true
#reload.period: 10s
modules:
enabled: true
path: /etc/customerbeat/modules.d/*.yml
#reload.enabled: true
#reload.period: 10s
#================================ General ======================================
name: "omitted"
#tags: ["888888"]
#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:
enabled: true
loadbalance: true
hosts: ["https://omitted:9200", "https://omitted:9200", "https://omitted:9200"]
#compression_level: 0
#escape_html: false
protocol: "https"
username: "elastic"
password: "changeme"
worker: 4
index: "888888-customerbeat"
max_retries: 3
bulk_max_size: 3200
backoff.init: 1s
backoff.max: 60s
timeout: 90
ssl.enabled: true
ssl.verification_mode: full
#ssl.supported_protocols: [TLSv1.3]
ssl.certificate_authorities: ["/etc/filebeat/certs/ca/omittedmonca.crt"]
ssl.certificate: "/etc/filebeat/certs/omitted.crt"
ssl.key: "/etc/filebeat/certs/omitted.key"
#ssl.key_passphrase: ''
#ssl.cipher_suites: []
#ssl.curve_types: []
#ssl.renegotiation: never
#================================= Paths ======================================
path.home: /usr/share/filebeat/
path.config: /etc/customerbeat/
path.data: /var/lib/customerbeat/data
path.logs: /var/log/customerbeat/
#================================ Keystore ====================================
# Location of the Keystore containing the keys and their sensitive values.
keystore.path: "${path.config}/beats.keystore"
#================================ Logging ======================================
logging.level: info
#logging.selectors: [ ]
logging.to_stderr: false
logging.to_syslog: false
logging.to_eventlog: false
logging.metrics.enabled: true
logging.metrics.period: 30s
logging.to_files: true
logging.files:
path: /var/log/customerbeat/
name: customerbeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 2
permissions: 0600
interval: 24h
rotateonstartup: False
logging.json: true
# ====================== Index Lifecycle Management (ILM) ======================
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "888888-customerbeat"
setup.ilm.pattern: "000001"
setup.ilm.policy_name: "888888-customerbeat_policy"
setup.ilm.policy_file: /etc/customerbeat/customerbeat_ilm.json
setup.template:
name: "888888-customerbeat"
pattern: "888888-*"
enabled: true
setup.ilm.overwrite: true
setup.dashboards:
enabled: true
#index: "888888-*"
setup.kibana.space.id: "888888"
#============================== X-Pack Monitoring ===============================
monitoring.enabled: false
#monitoring.cluster_uuid:
monitoring.elasticsearch:
hosts: ["https://omitted:9200", "https://omitted:9200", "https://omitted:9200"]
compression_level: 1
protocol: "https"
username: "elastic"
password: "changeme"
max_retries: 3
bulk_max_size: 50
backoff.init: 1s
backoff.max: 60s
timeout: 90
ssl.enabled: true
ssl.verification_mode: full
#ssl.supported_protocols: [TLSv1.3]
ssl.certificate_authorities: ["/etc/filebeat/certs/ca/omittedmonca.crt"]
ssl.certificate: "/etc/filebeat/certs/omitted.crt"
ssl.key: "/etc/filebeat/certs/omitted.key"
#ssl.key_passphrase: ''
#ssl.cipher_suites: []
#ssl.curve_types: []
#ssl.renegotiation: never
metrics.period: 1m
state.period: 10m
# ================================= Dashboards =================================
setup.kibana:
host: "https://omitted:5601"
protocol: "https"
username: "elastic"
password: "changeme "
path: ""
ssl.enabled: true
ssl.verification_mode: full
#ssl.supported_protocols: [TLSv1.3]
ssl.certificate_authorities: ["/etc/filebeat/certs/ca/omittedmonca.crt"]
ssl.certificate: "/etc/filebeat/certs/omitted.crt"
ssl.key: "/etc/filebeat/certs/omitted.key"
#================================ HTTP Endpoint ======================================
http.enabled: true
http.host: "omitted"
http.port: 5062
#============================= Process Security ================================
# Enable or disable seccomp system call filtering on Linux. Default is enabled.
seccomp.enabled: true
#================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: false
JGreene
(Jared Greene)
December 2, 2021, 1:06pm
4
Additionally, we are on 7.15.2 (Beats, Elastic, & Kibana). However, this appears to happen for previous versions as well.
system
(system)
Closed
December 30, 2021, 3:06pm
5
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.