HI!
I am running a filebeat version 8.5.3 daemonset in kubernetes cluster. I has configured autodiscover filestream input each namespace. it have error logs with ID already exists are often reported during run time. My autodiscover configuration is as follows:
filebeat.autodiscover:
providers:
- type: kubernetes
scope: node
namespace: test
templates:
- condition:
equals:
kubernetes.labels:
k8s-app: java
config:
- type: filestream
id: miaoyin-test-java
paths:
- /output/logs/test/miaoyin/${data.kubernetes.container.name}/*.log
prospector.scanner.exclude_files: ['\.gz$','.*-error\.log','gc.*\.log']
parsers:
- multiline:
type: pattern
pattern: '(?m)\d+-\d+-\d+\s\d+:\d+:\d+.\d+'
negate: true
match: after
- type: kubernetes
namespace: test01
templates:
- condition:
equals:
kubernetes.labels:
k8s-app: java
config:
- type: filestream
id: miaoyin-test01-java
paths:
- /output/logs/test01/miaoyin/${data.kubernetes.container.name}/*.log
prospector.scanner.exclude_files: ['\.gz$','.*-error\.log','gc.*\.log']
parsers:
- multiline:
type: pattern
pattern: '(?m)\d+-\d+-\d+\s\d+:\d+:\d+.\d+'
negate: true
match: after
The error logs is as follows:
{"log.level":"error","@timestamp":"2023-01-09T09:30:44.265Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":182},"message":"filestream input with ID 'miaoyin-test-java' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.266Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'filestream' starting","service.name":"filebeat","id":"miaoyin-test-java","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"miaoyin-test01-java","source_file":"filestream::miaoyin-test01-java::native::4722215-64512","path":"/output/logs/test01/miaoyin/sbds-live/sbds-live.log","state-id":"native::4722215-64512","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":294},"message":"Error while stopping harvester group: task failures\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file","service.name":"filebeat","id":"miaoyin-test01-java","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":126},"message":"Input 'filestream' stopped (goroutine)","service.name":"filebeat","id":"miaoyin-test01-java","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":134},"message":"Input 'filestream' stopped (runner)","service.name":"filebeat","id":"miaoyin-test01-java","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-09T09:30:44.646Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":182},"message":"filestream input with ID 'miaoyin-test01-java' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}