Filebeat autodiscover filestream input have error logs with ID already exists

Elastic Stack Beats
1

HI!
I am running a filebeat version 8.5.3 daemonset in kubernetes cluster. I has configured autodiscover filestream input each namespace. it have error logs with ID already exists are often reported during run time. My autodiscover configuration is as follows:

filebeat.autodiscover:
      providers:
        - type: kubernetes
          scope: node
          namespace: test
          templates:
            - condition:
                equals:
                  kubernetes.labels:
                    k8s-app: java
              config:
                - type: filestream
                  id: miaoyin-test-java
                  paths:
                    - /output/logs/test/miaoyin/${data.kubernetes.container.name}/*.log
                  prospector.scanner.exclude_files: ['\.gz$','.*-error\.log','gc.*\.log']
                  parsers:
                    - multiline:
                        type: pattern
                        pattern: '(?m)\d+-\d+-\d+\s\d+:\d+:\d+.\d+'
                        negate: true
                        match: after
        - type: kubernetes
          namespace: test01
          templates:
            - condition:
                equals:
                  kubernetes.labels:
                    k8s-app: java
              config:
                - type: filestream
                  id: miaoyin-test01-java
                  paths:
                    - /output/logs/test01/miaoyin/${data.kubernetes.container.name}/*.log
                  prospector.scanner.exclude_files: ['\.gz$','.*-error\.log','gc.*\.log']
                  parsers:
                    - multiline:
                        type: pattern
                        pattern: '(?m)\d+-\d+-\d+\s\d+:\d+:\d+.\d+'
                        negate: true
                        match: after

The error logs is as follows:

{"log.level":"error","@timestamp":"2023-01-09T09:30:44.265Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":182},"message":"filestream input with ID 'miaoyin-test-java' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.266Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'filestream' starting","service.name":"filebeat","id":"miaoyin-test-java","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"miaoyin-test01-java","source_file":"filestream::miaoyin-test01-java::native::4722215-64512","path":"/output/logs/test01/miaoyin/sbds-live/sbds-live.log","state-id":"native::4722215-64512","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":294},"message":"Error while stopping harvester group: task failures\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file","service.name":"filebeat","id":"miaoyin-test01-java","prospector":"file_prospector","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":126},"message":"Input 'filestream' stopped (goroutine)","service.name":"filebeat","id":"miaoyin-test01-java","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-01-09T09:30:44.267Z","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":134},"message":"Input 'filestream' stopped (runner)","service.name":"filebeat","id":"miaoyin-test01-java","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-09T09:30:44.646Z","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":182},"message":"filestream input with ID 'miaoyin-test01-java' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.