I have a trouble with events dublicates in elasticsearch.
I used filebeat to aggregate events from logback logs. It's config:
filebeat.inputs:
- type: log
enabled: true
paths:
- /#/*/#/*/#/logs/*.log
- /#/*/#/*/#/logs/catalina.out
document_type: container
#exclude_lines: ['^DBG']
#include_lines: ['^ERR', '^WARN']
#exclude_files: ['.gz$']
multiline.pattern: '^\['
multiline.negate: false
multiline.match: after
multiline.max_lines: 10000
ignore_older: 3m
close_timeout: 2m
clean_inactive: 2h
scan_frequency: 1s
close_renamed: true
close_removed: true
clean_removed: true
fields_under_root: true
harvester_limit: 0
- type: log
enabled: true
paths:
- /#/*/#/logs/*.log
document_type: service
#exclude_lines: ['^DBG']
#include_lines: ['^ERR', '^WARN']
#exclude_files: ['.gz$']
multiline.pattern: '^\['
multiline.negate: false
multiline.match: after
multiline.max_lines: 10000
ignore_older: 3m
close_timeout: 2m
clean_inactive: 2h
scan_frequency: 1s
close_renamed: true
close_removed: true
clean_removed: true
fields_under_root: true
harvester_limit: 0
tags: ["#####"]
#==================== Elasticsearch template setting ==========================
setup.template:
name: 'deploy'
pattern: '*'
fields: "/etc/filebeat/fields.yml"
overwrite: true
enabled: true
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
#============================== Dashboards =====================================
setup.dashboards.enabled: true
setup.dashboards.index: "10288-*"
setup.dashboards.url:
#============================== Kibana =====================================
setup.kibana:
host: "deploy-elk1.mhd.local:5601"
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
hosts: ["###:9200"]
timeout: 600
# protocol: "https"
# username: "elastic"
# password: "#####"
max_retries: 0
backoff.init: 600s
backoff.max: 600s
index: "10288-%{+yyyy.MM.dd-HH.mm}"
#================================ Logging =====================================
logging.level: info
#logging.selectors: ["*"]
Errors in log:
2018-10-10T06:34:17.781-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
2018-10-10T06:34:17.939-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
2018-10-10T06:34:33.712-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
2018-10-10T06:34:52.725-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
2018-10-10T06:35:01.709-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
2018-10-10T06:35:02.886-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
2018-10-10T06:35:28.325-0400 ERROR registrar/registrar.go:346 Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...
But I have 2 events by every line in log files. What can be wrong?
Filebeat collecting information about services starts, only first 2 minutes and killing after by script.
