Filestream input ID without ID might lead to data duplication

jean.bissonnette · April 14, 2022, 8:58pm

Filebeat 8.1.2 is installed on a windows server with web applications generating log files and using IIS.

I have this error when I start filebeat via windows service.
{"log.level":"error","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":176},"message":"filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat","service.name":"filebeat","ecs.version":"1.6.0"}**
**{"log.level":"error","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":181},"message":"filestream input with ID '' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}

It says it comes from file.name":"input-logfile/manager.go". But there is no such file in the environment.

What is the cause of this problem?

I have also this warning:
{"log.level":"warn","@timestamp":"2022-04-14T15:21:41.492-0400","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}

The certificate is the pem file created in elasticsearch node with the command:
./bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12
I took the pem file in Kibana folder of the zip file created and included it in the filebeat.yml configuration file.

output.elasticsearch:
  hosts: ["172.16.82.115:9200"]
  protocol: "https"
  username: "filebeat_writer"
  password: "XXX"

  ssl.certificate_authorities: ${path.config}/elasticsearch-ca.pem
  ssl.verification_mode: "certificate"

This is according to instructions found in elasticsearch documentation in beats section of Set up basic security for the Elastic Stack plus secured HTTPS traffic | Elasticsearch Guide [8.1] | Elastic

image864×375 14.4 KB

What is the cause of the warning and what should I change?

Following is the filebeat logs.

{"log.level":"info","@timestamp":"2022-04-14T15:21:38.454-0400","log.origin":{"file.name":"instance/beat.go","file.line":669},"message":"Home path: [D:\\psoft\\Logging\\filebeat-8.1.2-windows-x86_64] Config path: [D:\\psoft\\Logging\\filebeat-8.1.2-windows-x86_64] Data path: [C:\\ProgramData\\filebeat-8.1.2] Logs path: [C:\\ProgramData\\filebeat\\logs-8.1.2]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:38.454-0400","log.origin":{"file.name":"instance/beat.go","file.line":677},"message":"Beat ID: 36ae1dd5-45cc-4ba6-b7b2-a20ade2ae09f","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-04-14T15:21:41.477-0400","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":80},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.478-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1047},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"D:\\psoft\\Logging\\filebeat-8.1.2-windows-x86_64","data":"C:\\ProgramData\\filebeat-8.1.2","home":"D:\\psoft\\Logging\\filebeat-8.1.2-windows-x86_64","logs":"C:\\ProgramData\\filebeat\\logs-8.1.2"},"type":"filebeat","uuid":"36ae1dd5-45cc-4ba6-b7b2-a20ade2ae09f"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.478-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1056},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"6118f25235a52a7f0c4937a0a309e380c92d8119","libbeat":"8.1.2","time":"2022-03-29T22:33:32.000Z","version":"8.1.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.478-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1059},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.17.6"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.478-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1063},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2022-03-20T17:35:15.68-04:00","name":"SRV-TICO-WEB01","ip":["172.16.82.20/23","172.16.82.21/23","172.16.82.22/23","172.16.82.23/23","172.16.82.24/23","172.16.82.153/23","::1/128","127.0.0.1/8"],"kernel_version":"10.0.20348.587 (WinBuild.160101.0800)","mac":["00:50:56:a8:92:e9"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2022 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"20348.587"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"e333e945-44ea-4ca8-8b3f-2a23fd882134"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.478-0400","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1092},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\WINDOWS\\system32","exe":"D:\\psoft\\Logging\\filebeat-8.1.2-windows-x86_64\\filebeat.exe","name":"filebeat.exe","pid":12596,"ppid":816,"start_time":"2022-04-14T15:21:38.218-0400"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.478-0400","log.origin":{"file.name":"instance/beat.go","file.line":323},"message":"Setup Beat: filebeat; Version: 8.1.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-04-14T15:21:41.492-0400","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.492-0400","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: https://172.16.82.115:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.492-0400","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: SRV-TICO-WEB01","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.498-0400","log.origin":{"file.name":"fileset/modules.go","file.line":103},"message":"Enabled modules/filesets:  ()","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.499-0400","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":105},"message":"elasticsearch url: https://172.16.82.115:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.499-0400","log.origin":{"file.name":"instance/beat.go","file.line":489},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.500-0400","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":142},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.502-0400","log.origin":{"file.name":"memlog/store.go","file.line":119},"message":"Loading data file of 'C:\\ProgramData\\filebeat-8.1.2\\registry\\filebeat' succeeded. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.505-0400","log.origin":{"file.name":"memlog/store.go","file.line":124},"message":"Finished loading transaction log file for 'C:\\ProgramData\\filebeat-8.1.2\\registry\\filebeat'. Active transaction id=83","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.505-0400","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.505-0400","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 7","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.505-0400","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.fields.env filebeat.inputs.0.fields.host filebeat.inputs.0.fields.log_type filebeat.inputs.0.fields.provider filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-04-14T15:21:41.505-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":176},"message":"filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.509-0400","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":284},"message":"Attempting to connect to Elasticsearch version 8.1.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.510-0400","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":232},"message":"Successfully connected to X-Pack Monitoring endpoint.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.510-0400","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":246},"message":"Start monitoring stats metrics snapshot loop with period 10s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.510-0400","log.logger":"monitoring","log.origin":{"file.name":"elasticsearch/elasticsearch.go","file.line":246},"message":"Start monitoring state metrics snapshot loop with period 1m0s.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.511-0400","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 8876244467491110051)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.511-0400","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.1.enabled filebeat.inputs.1.fields.env filebeat.inputs.1.fields.host filebeat.inputs.1.fields.log_type filebeat.inputs.1.fields.provider filebeat.inputs.1.parsers.0.multiline.match filebeat.inputs.1.parsers.0.multiline.negate filebeat.inputs.1.parsers.0.multiline.pattern filebeat.inputs.1.parsers.0.multiline.type filebeat.inputs.1.paths.0 filebeat.inputs.1.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.511-0400","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":111},"message":"Input filestream starting","service.name":"filebeat","id":"7B2EC15098557CA3","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-04-14T15:21:41.511-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":176},"message":"filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-04-14T15:21:41.511-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":181},"message":"filestream input with ID '' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.511-0400","log.logger":"file_watcher","log.origin":{"file.name":"filestream/fswatch.go","file.line":138},"message":"Start next scan","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 17796714877607627319)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.2.enabled filebeat.inputs.2.fields.env filebeat.inputs.2.fields.host filebeat.inputs.2.fields.log_type filebeat.inputs.2.fields.provider filebeat.inputs.2.parsers.0.multiline.match filebeat.inputs.2.parsers.0.multiline.negate filebeat.inputs.2.parsers.0.multiline.pattern filebeat.inputs.2.parsers.0.multiline.type filebeat.inputs.2.paths.0 filebeat.inputs.2.type]","service.name":"filebeat","ecs.version":"1.6.0"}
**{"log.level":"error","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":176},"message":"filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat","service.name":"filebeat","ecs.version":"1.6.0"}**
**{"log.level":"error","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"input","log.origin":{"file.name":"input-logfile/manager.go","file.line":181},"message":"filestream input with ID '' already exists, this will lead to data duplication, please use a different ID","service.name":"filebeat","ecs.version":"1.6.0"}**
{"log.level":"info","@timestamp":"2022-04-14T15:21:41.515-0400","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":111},"message":"Input filestream starting","service.name":"filebeat","id":"F6FAA1E3E0B64A37","ecs.version":"1.6.0"}

Thank you

TiagoQueiroz · April 20, 2022, 1:24pm

Hi @jean.bissonnette,

You're describing two very different things, so let's go one by one.

1. filestream input ID without ID might lead to data duplication, please add an ID and restart Filebeat

It seems you're running Filebeat stand alone using the filestream input, right?

The error message you're seeing is because on your configuration file there are more than one filestream input without an ID set. To fix that just set a unique ID for each filestream input on your configuration file. Something like this:

filebeat.inputs:

- type: filestream
  enabled: true
  id: "foo-bar"
  paths:
    - /foo/bar*.log

- type: filestream
  enabled: true
  id: "something-else"
  paths:
    - /somehting/else/*.log

The field file.name refers to Filebeat's source code, the file and line of code that issued that log message, hence you won't find it in your environment

2. Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed.

This is a warning that is printed every time Filebeat starts, it is printed regardless of the certificates being used. It is there to warn about this change in certificates validation that will be introduced at some point, thus rendering some old certificates invalid.

If you followed our documentation and used the tools shipped with Elasticsearch to generate the certificates you don't have to worry about anything.

jean.bissonnette · April 21, 2022, 9:50pm

Thank you for the explanations.
I initially followed this example found in the elasticsearch documentation of this page: filestream input | Filebeat Reference [8.1] | Elastic.

image907×607 29.5 KB

This example will fail because there is no ID specified.
Would be nice to adjust documentation to help other elasticsearch users not fall in the same quest as I did.

The impact of not providing an ID per filestream affects also the filebeat windows service.
We can start the service but we cannot stop the service. See what happens:

Even if you kill the service, the problem remains. You have to uninstall the service and install again.

TiagoQueiroz · April 25, 2022, 10:53am

Thanks for catching this! We do have a documentation fix in master: filestream input | Filebeat Reference [master] | Elastic but it seems it haven't made it to all versions yet.
I'll make sure the older versions also get this update.

That's interesting. As far as I know we haven't come across this Filebeat not stopping due missing/duplicated IDs in filestream.

Could you provide your configuration (don't forget to redact any private/sensitive information)? I'd like to investigate it.

If you still have the logs from when Filebeat didn't stop that would also be helpful.

system · May 23, 2022, 12:53pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.