Filebeat raise error:filestream input id already exist

Elastic Stack Beats
,
1

I install filebeat-8.1.2 on centos 7, then use filestream input to read log files:

- type: filestream
  id: ro-server-run-log
  paths: ['/data/ro_jplive/log/*-*-*-????-??-??-??.log'] 
  prospector.scanner.exclude_files: ['/data/ro_jplive/log/commonlib.*', '/data/ro_jplive/log/.+-perf.+']
  ignore_older: 10m
  close.on_state_change.inactive: 2m
  clean_inactive: 15m
  index: ro-server

when I star filebeat.service , beat log raise error log:

{"log.level":"error","@timestamp":"2022-05-20T10:44:48.068+0800","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":181},"message":"filestream input with ID 'ro-server-run-log' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}

I didn't find a solution to this problem

1 Like
2

You must have unique IDs. ro-server-run-log exists somewhere else, change it.

3

I have only one configuration file, named ro-server.yml, and there is only one input configuration in it

4

If is possible rename it temporarily.

5

My filebeat.yml is as follows:

name: "log-center-filebeat"
max_procs: 2
fields:
  owner:
    vendor: "huanle"
    region: "shanghai"
    zone: shanghai-a
    project: "ro"
    app: "log-center"
  os:
    private_ip: 10.4.7.57
    public_ip: 118.39.23.25
    kernel: "3.10.0-1160.45.1.el7.x86_64"
    architecture: "x86_64"
    distribution: "CentOS"
    version: "7.9"
fields_under_root: true
filebeat.shutdown_timeout: 5s
logging:
  level: error
  to_files: true
  files:
    path: /opt/logs/log-center-filebeat
    name: filebeat.log
    keepfiles: 5
    rotateonstartup: false
    permissions: 0644
filebeat.config.inputs:
  enabled: true
  path: inputs.d/*.yml

setup:
  ilm.enabled: false
  template:
    name: system-template
    pattern: "system*"

output.console:
  pretty: true
  enabled: False

output.file:
  enabled: True
  path: /opt/logs/log-center-filebeat
  filename: filebeat.output
  rotate_every_kb: 40960
  permissions: 0644

The inputs.d directory has only one config:

[root@centos-7-57 log-center-filebeat]# tree inputs.d/
inputs.d/
└── ro-server.yml

0 directories, 1 file

The ro-server.yml :

- type: filestream
  id: "ro-server-run-log"
  paths: ['/data/ro_jplive/log/*-*-*-????-??-??-??.log'] 
  prospector.scanner.exclude_files: ['/data/ro_jplive/log/commonlib.*', '/data/ro_jplive/log/.+-perf.+']
  ignore_older: 10m
  close.on_state_change.inactive: 2m
  clean_inactive: 15m
  index: ro-server
  parsers:
  - multiline:
      type: pattern
      pattern: '^\['
      negate: true
      match: after
  processors:
  - dissect:
      tokenizer: '[%{level}] %{time},%{millisecond} %{service} - %{function}(%{code}) %{msg}'
      field: "message"
  - dissect:
      tokenizer: '/data/ro_jplive/log/%{instance_name}-%{server_id}-%{instance_id}-%{date_str}.log'
      field: "log.file.path"
  - dissect:
      tokenizer: '/data/ro_jplive/log/%{log_file}'
      field: "log.file.path"
  - timestamp:
      field: "dissect.time"
      timezone: "Local"
      layouts:
      - "2006-01-02 15:04:05"
      when:
        has_fields: "dissect.time"
  - drop_fields:
      fields: ["log", "dissect.millisecond", "dissect.date_str", "dissect.time", "ecs"]
      ignore_missing: true

I remove filebeat data directory ,then start filebeat.server . The log shows the following error:

{"log.level":"error","@timestamp":"2022-05-20T14:18:47.953+0800","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":181},"message":"filestream input with ID 'ro-server-run-log' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}

Is there something wrong with my configuration?

6

filebeat v8.1.2 & v8.1.3 filestream raise error. Same configuration, v8.1.1 & v8.1.0 no errors. Maybe a bug?

7

Try to set clean_removed: false

link

8

I do not see anything wrong with you configuration. It seems that the logging of the filestream is a bit too dramatic. What you see is not an error, but rather a warning if you configured the same ID for different inputs. We should definitely reword this.

1 Like
9

This is a bug, unfortunately. I've opened an issue: `filestream` input returns error when an existing input is reloaded with the same ID · Issue #31767 · elastic/beats · GitHub

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.