# Filebeat fingerprint excessive logs **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 2, 2024, 6:21pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440 "2024-02-02T18:21:22Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 2, 2024, 6:21pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/1 "2024-02-02T18:21:23Z")
Hi We have turned on fingerprinting in our lab. We see a lot of these logs. Is there a way to disabled those 0 bytes logs? Or make fingerprint ignore those files? ```auto {"log.level":"warn","@timestamp":"2024-02-02T18:16:22.244Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/puppet/masterhttp.log\": filesize of \"/var/log/puppet/masterhttp.log\" is 0 bytes, expected at least 1024 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} ``` Portion of our config ```auto filebeat.autodiscover: providers: - add_resource_metadata: cronjob: false deployment: false hints.default_config: close.on_state_change.renamed: true id: filestream-kubernetes-pod-${data.kubernetes.container.id} parsers: - container: null paths: - /var/log/containers/*-${data.kubernetes.container.id}.log prospector.scanner.symlinks: true type: filestream hints.enabled: true host: ${NODE_NAME} type: kubernetes filebeat.inputs: - close.reader.after_interval: 5m fingerprint: enabled: true length: 1024 offset: 0 id: wra-filestream-id paths: - /var/log/*.log - /var/log/messages - /var/log/syslog - /var/log/**/*.log prospector.scanner.exclude_files: - ^/var/log/containers/ - ^/var/log/pods/ prospector.scanner.fingerprint.enabled: true type: filestream ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 4, 2024, 12:50am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/2 "2024-02-04T00:50:12Z")
@kbujold_wr It seems perhaps you have solved this .. can you share your result / fix or perhaps what you learned so others can learn? ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 6, 2024, 8:05pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/3 "2024-02-06T20:05:23Z")
A fixed has not been found sorry for the confusion. Is there a way to turn this logging off when a file is zero bytes? This will flood the filebeat logs. ```auto {"log.level":"warn","@timestamp":"2024-02-06T19:58:27.604Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:58:37.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:58:47.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:58:57.804Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:59:07.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:59:17.807Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:59:27.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:59:37.806Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:59:47.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T19:59:57.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:00:07.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:00:17.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:00:27.704Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:00:37.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:00:47.904Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:00:57.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} {"log.level":"warn","@timestamp":"2024-02-06T20:01:07.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"} ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 6, 2024, 8:06pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/4 "2024-02-06T20:06:32Z")
Perhaps look at the logging configuration perhaps set `logging.level: error` > **[Configure logging | Filebeat Reference \[8.12\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html)** ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 6, 2024, 8:12pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/5 "2024-02-06T20:12:01Z")
> [@stephenb](#): > > Perhaps look at the logging configuration perhaps set > > `logging.level: error` We cannot set our logging to the error level. This code goes in production and warning level is required. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 6, 2024, 8:29pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/6 "2024-02-06T20:29:46Z")
> [@kbujold\_wr](#): > > We cannot set our logging to the error level. This code goes in production and warning level is required. So a quick look at the docs [here](https://www.elastic.co/guide/en/beats/filebeat/8.12/filebeat-input-filestream.html#filebeat-input-filestream-scan-fingerprint) ... 1st, I don't think you actually have the fingerprint config correct in the YML. The docs look weird, but I think this is correct. You are just getting defaults the way you have it. ```auto prospector.scanner.fingerprint: enabled: true length: 1024 offset: 0 ``` There does not seem to be a way to ignore 0 length files for fingerprint. Question are you trying to get rid of these logs being written to the filesystem or Elasticsearch? ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 6, 2024, 8:47pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/7 "2024-02-06T20:47:05Z")
@stephenb I have since updated to ```auto filebeat.autodiscover: providers: - add_resource_metadata: cronjob: false deployment: false hints.default_config: close.on_state_change.renamed: true id: filestream-kubernetes-pod-${data.kubernetes.container.id} parsers: - container: null paths: - /var/log/containers/*-${data.kubernetes.container.id}.log prospector.scanner.symlinks: true type: filestream hints.enabled: true host: ${NODE_NAME} type: kubernetes filebeat.inputs: - close.reader.after_interval: 5m file_identity.fingerprint: null fingerprint: enabled: true length: 64 offset: 0 id: wra-filestream-id paths: - /var/log/*.log - /var/log/messages - /var/log/syslog - /var/log/**/*.log prospector.scanner.exclude_files: - ^/var/log/containers/ - ^/var/log/pods/ prospector.scanner.fingerprint: enabled: true length: 64 offset: 0 type: filestream ``` The problem is the zero bytes files will be logged at repeated interval until there are not zero anymore. This is an example in one of our labs. These files below have generated 260 logs each so date with the filebeat pod running for a short time. ```auto kubectl -n monitor logs mon-filebeat-2s74v | grep "is 0 bytes" | awk '{print $10}' | sort | uniq -c Defaulted container "filebeat" out of: filebeat, a-beat-setenv (init), b-security-setenv (init), c-beat-setup (init) 260 \"/var/log/horizon/gunicorn.log\": 260 \"/var/log/horizon/horizon.log\": 260 \"/var/log/kubernetes/audit/audit.log\": 260 \"/var/log/ldapscripts.log\": 260 \"/var/log/lighttpd-access.log\": 260 \"/var/log/patching-api.log\": 260 \"/var/log/postgresql/postgresql-13-main.log\": 260 \"/var/log/puppet/masterhttp.log\": 260 \"/var/log/rabbitmq/log/crash.log\": 260 \"/var/log/sm-trap.log\": 260 \"/var/log/sssd/sssd_pam.log\": ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 6, 2024, 9:42pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/8 "2024-02-06T21:42:01Z")
@kbujold_wr I think I understand the problem, but I am asking specific question which you have not answered Where are you seeing the log messages that you want to get rid of a) In Elasticsearch or b) On the filesystem Side Note **NOTE: FIxed** > [@kbujold\_wr](#): > > ```auto > - close.reader.after_interval: 5m > > # Not Correct... > file_identity.fingerprint: null > > # Not Correct... > fingerprint: enabled: true length: 64 offset: 0 .... > # Correct > prospector.scanner.fingerprint: > enabled: true > length: 64 > offset: 0 > > # OR Correct > prospector.scanner.fingerprint.enabled: true > prospector.scanner.fingerprint.length: 64 > prospector.scanner.fingerprint.offset: 0 > > ``` ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 7, 2024, 12:51am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/9 "2024-02-07T00:51:02Z")
a) The log are coming from the filebeat container. ELK is running on kubernetes. b) The ~ get translated into null in the yml file. Is the intent to use the ~ with quotes? So the value is actually a string? Otherwise its intent is null [What is the purpose of tilde character ~ in YAML? - Stack Overflow](https://stackoverflow.com/questions/51990175/what-is-the-purpose-of-tilde-character-in-yaml) ```auto filebeat.inputs: - close.reader.after_interval: 5m file_identity.fingerprint: '~' id: wra-filestream-id paths: - /var/log/*.log - /var/log/messages - /var/log/syslog - /var/log/**/*.log prospector.scanner.exclude_files: - ^/var/log/containers/ - ^/var/log/pods/ prospector.scanner.fingerprint: enabled: true length: 64 offset: 0 type: filestream ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 7, 2024, 1:52am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/10 "2024-02-07T01:52:00Z")
Hi @kbujold_wr I am trying to solve your original question about "muting/dropping" the logs are you still interested in that? If so, please answer my question above ... otherwise, I can move on to questions from other users... > [@stephenb](#): > > Where are you seeing the log messages that you want to get rid of > > a) In Elasticsearch > > or > > b) On the filesystem Regarding the .yml from our docs reference yes sorry I think that is a bug / or maybe `~` means fill in the values ... yes kinda weird maybe a doc error or different convention ... I will check I fixed the code above so it does not confuse anyone else, I should have check the reference first. but here is the reference.... below which you can always look at that.... Please refer to the reference .yml > **[filebeat.reference.yml | Filebeat Reference \[8.12\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/8.12/filebeat-reference-yml.html)** ```auto # If enabled, instead of relying on the device ID and inode values when comparing files, # compare hashes of the given byte ranges in files. A file becomes an ingest target # when its size grows larger than offset+length (see below). Until then it's ignored. #prospector.scanner.fingerprint.enabled: false # If fingerprint mode is enabled, sets the offset from the beginning of the file # for the byte range used for computing the fingerprint value. #prospector.scanner.fingerprint.offset: 0 # If fingerprint mode is enabled, sets the length of the byte range used for # computing the fingerprint value. Cannot be less than 64 bytes. #prospector.scanner.fingerprint.length: 1024 ``` So yes it should look like ```auto # If enabled, instead of relying on the device ID and inode values when comparing files, # compare hashes of the given byte ranges in files. A file becomes an ingest target # when its size grows larger than offset+length (see below). Until then it's ignored. prospector.scanner.fingerprint.enabled: true # If fingerprint mode is enabled, sets the offset from the beginning of the file # for the byte range used for computing the fingerprint value. prospector.scanner.fingerprint.offset: 0 # If fingerprint mode is enabled, sets the length of the byte range used for # computing the fingerprint value. Cannot be less than 64 bytes. prospector.scanner.fingerprint.length: 1024 ``` ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 7, 2024, 3:08am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/11 "2024-02-07T03:08:37Z")
Hi @stephenb Yes we would like to be able to mute those logs. The logs come from elastic so a). There are generated by filebeat. To view them on our system we use ```auto kubectl -n monitor logs mon-filebeat-stmsv ``` They can be viewed on the Discover page as well ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/3/a3c3dce708d12227d2fa570a6857d6aedc3f2e15.png) ---
**Author:** ![El-k](https://avatars.discourse-cdn.com/v4/letter/e/b5e925/32.png) [@El-k](https://discuss.elastic.co/u/El-k)\ **Post date:** [February 12, 2024, 8:07am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/12 "2024-02-12T08:07:28Z")
Hi, @kbujold_wr Were you able to solve this? I'm facing the same problem... but one thing: I get also warnings about files smaller than 1024 bytes but not 0, like: ```auto [warn] cannot create a file descriptor for an ingest target "/var/log/containers/alertmanager-0_gmp-system_config-reloader-783b41888d5054437d89933748846aab311545e17b51c36f64b.log": filesize of "/var/log/containers/alertmanager-0_gmp-system_config-reloader-783b41888d5054437d89933748846aab311545e17b51c36f64b.log" is 999 bytes, expected at least 1024 bytes for fingerprinting ``` ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [February 12, 2024, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/13 "2024-02-12T15:03:29Z")
I think the only way you are going to be able to mute those specific logs is with an ingest pipeline and a `drop` processor with the correct condition... ---
**Author:** ![kbujold\_wr](https://avatars.discourse-cdn.com/v4/letter/k/9fc29f/32.png) [@kbujold\_wr](https://discuss.elastic.co/u/kbujold_wr)\ **Post date:** [February 13, 2024, 7:58pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/14 "2024-02-13T19:58:18Z")
> [@stephenb](#): > > ingest pipeline Do you have an example on how and where to set this up? Ultimately these logs should probably be set to info. ---
**Author:** ![GEownt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/geownt/32/131957_2.png) [@GEownt](https://discuss.elastic.co/u/GEownt)\ **Post date:** [February 23, 2024, 6:14am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/15 "2024-02-23T06:14:11Z")
@kbujold_wr We had the same problem where the 0 byte logs were spamming our logs with warnings. I just replaced the log.level for these logs to info in filebeat: ```auto filebeat.autodiscover: providers: - type: kubernetes node: ${NODE_NAME} ... processors: - replace: when: contains: message: "is 0 bytes, expected at least 64 bytes for fingerprinting" fields: - field: "log.level" pattern: "warn" replacement: "info" ``` This would be the easiest way I know to set the log.level for 0 bytes to info. U can use `regexp` or anything else to be more clear on which event should be set to `info` level but in our case we don´t have any other logs with "64 bytes for fingerprinting" so the processor `contains` is good for us. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 22, 2024, 8:14am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-excessive-logs/352440/16 "2024-03-22T08:14:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.