Filebeat fingerprint excessive logs

[kbujold_wr]

Hi

We have turned on fingerprinting in our lab.

We see a lot of these logs. Is there a way to disabled those 0 bytes logs? Or make fingerprint ignore those files?

{"log.level":"warn","@timestamp":"2024-02-02T18:16:22.244Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/puppet/masterhttp.log\": filesize of \"/var/log/puppet/masterhttp.log\" is 0 bytes, expected at least 1024 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}

Portion of our config

filebeat.autodiscover:
  providers:
  - add_resource_metadata:
      cronjob: false
      deployment: false
    hints.default_config:
      close.on_state_change.renamed: true
      id: filestream-kubernetes-pod-${data.kubernetes.container.id}
      parsers:
      - container: null
      paths:
      - /var/log/containers/*-${data.kubernetes.container.id}.log
      prospector.scanner.symlinks: true
      type: filestream
    hints.enabled: true
    host: ${NODE_NAME}
    type: kubernetes
filebeat.inputs:
- close.reader.after_interval: 5m
  fingerprint:
    enabled: true
    length: 1024
    offset: 0
  id: wra-filestream-id
  paths:
  - /var/log/*.log
  - /var/log/messages
  - /var/log/syslog
  - /var/log/**/*.log
  prospector.scanner.exclude_files:
  - ^/var/log/containers/
  - ^/var/log/pods/
  prospector.scanner.fingerprint.enabled: true
  type: filestream

[stephenb (Stephen Brown)]

@kbujold_wr It seems perhaps you have solved this .. can you share your result / fix or perhaps what you learned so others can learn?

[kbujold_wr]

A fixed has not been found sorry for the confusion.
Is there a way to turn this logging off when a file is zero bytes? This will flood the filebeat logs.

{"log.level":"warn","@timestamp":"2024-02-06T19:58:27.604Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:58:37.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:58:47.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:58:57.804Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:59:07.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:59:17.807Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:59:27.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:59:37.806Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:59:47.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T19:59:57.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:00:07.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:00:17.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:00:27.704Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:00:37.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:00:47.904Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:00:57.509Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-02-06T20:01:07.508Z","log.logger":"scanner","log.origin":{"file.name":"filestream/fswatch.go","file.line":388},"message":"cannot create a file descriptor for an ingest target \"/var/log/horizon/horizon.log\": filesize of \"/var/log/horizon/horizon.log\" is 0 bytes, expected at least 64 bytes for fingerprinting","service.name":"filebeat","ecs.version":"1.6.0"}

[stephenb (Stephen Brown)]

Perhaps look at the logging configuration perhaps set

logging.level: error

Configure logging | Filebeat Reference [8.12] | Elastic

[kbujold_wr]

@stephenb

Perhaps look at the logging configuration perhaps set

logging.level: error

We cannot set our logging to the error level. This code goes in production and warning level is required.

[stephenb (Stephen Brown)]

@kbujold_wr

We cannot set our logging to the error level. This code goes in production and warning level is required.

So a quick look at the docs here ...

1st, I don't think you actually have the fingerprint config correct in the YML. The docs look weird, but I think this is correct. You are just getting defaults the way you have it.

prospector.scanner.fingerprint:
  enabled: true
  length: 1024
  offset: 0

There does not seem to be a way to ignore 0 length files for fingerprint.

Question are you trying to get rid of these logs being written to the filesystem or Elasticsearch?

[kbujold_wr]

@stephenb

I have since updated to

filebeat.autodiscover:
  providers:
  - add_resource_metadata:
      cronjob: false
      deployment: false
    hints.default_config:
      close.on_state_change.renamed: true
      id: filestream-kubernetes-pod-${data.kubernetes.container.id}
      parsers:
      - container: null
      paths:
      - /var/log/containers/*-${data.kubernetes.container.id}.log
      prospector.scanner.symlinks: true
      type: filestream
    hints.enabled: true
    host: ${NODE_NAME}
    type: kubernetes
filebeat.inputs:
- close.reader.after_interval: 5m
  file_identity.fingerprint: null
  fingerprint:
    enabled: true
    length: 64
    offset: 0
  id: wra-filestream-id
  paths:
  - /var/log/*.log
  - /var/log/messages
  - /var/log/syslog
  - /var/log/**/*.log
  prospector.scanner.exclude_files:
  - ^/var/log/containers/
  - ^/var/log/pods/
  prospector.scanner.fingerprint:
    enabled: true
    length: 64
    offset: 0
  type: filestream

The problem is the zero bytes files will be logged at repeated interval until there are not zero anymore.

This is an example in one of our labs. These files below have generated 260 logs each so date with the filebeat pod running for a short time.

kubectl -n monitor logs  mon-filebeat-2s74v | grep "is 0 bytes" | awk '{print $10}' | sort |  uniq -c
Defaulted container "filebeat" out of: filebeat, a-beat-setenv (init), b-security-setenv (init), c-beat-setup (init)
    260 \"/var/log/horizon/gunicorn.log\":
    260 \"/var/log/horizon/horizon.log\":
    260 \"/var/log/kubernetes/audit/audit.log\":
    260 \"/var/log/ldapscripts.log\":
    260 \"/var/log/lighttpd-access.log\":
    260 \"/var/log/patching-api.log\":
    260 \"/var/log/postgresql/postgresql-13-main.log\":
    260 \"/var/log/puppet/masterhttp.log\":
    260 \"/var/log/rabbitmq/log/crash.log\":
    260 \"/var/log/sm-trap.log\":
    260 \"/var/log/sssd/sssd_pam.log\":

[stephenb (Stephen Brown)]

@kbujold_wr

I think I understand the problem, but I am asking specific question which you have not answered

Where are you seeing the log messages that you want to get rid of

a) In Elasticsearch

or

b) On the filesystem

Side Note

NOTE: FIxed

@kbujold_wr

- close.reader.after_interval: 5m

# Not Correct... 
  file_identity.fingerprint: null

# Not Correct... 
  fingerprint:  <!- Not Needed 
    enabled: true  <!- Not Needed 
    length: 64  <!- Not Needed 
    offset: 0  <!- Not Needed 
....
# Correct
  prospector.scanner.fingerprint:
    enabled: true
    length: 64
    offset: 0

# OR Correct
  prospector.scanner.fingerprint.enabled: true
  prospector.scanner.fingerprint.length: 64
  prospector.scanner.fingerprint.offset: 0

[kbujold_wr]

a) The log are coming from the filebeat container. ELK is running on kubernetes.

b) The ~ get translated into null in the yml file. Is the intent to use the ~ with quotes? So the value is actually a string? Otherwise its intent is null What is the purpose of tilde character ~ in YAML? - Stack Overflow

filebeat.inputs:
- close.reader.after_interval: 5m
  file_identity.fingerprint: '~'
  id: wra-filestream-id
  paths:
  - /var/log/*.log
  - /var/log/messages
  - /var/log/syslog
  - /var/log/**/*.log
  prospector.scanner.exclude_files:
  - ^/var/log/containers/
  - ^/var/log/pods/
  prospector.scanner.fingerprint:
    enabled: true
    length: 64
    offset: 0
  type: filestream

[stephenb (Stephen Brown)]

Hi @kbujold_wr

I am trying to solve your original question about "muting/dropping" the logs are you still interested in that?

If so, please answer my question above ... otherwise, I can move on to questions from other users...

@stephenb

Where are you seeing the log messages that you want to get rid of

a) In Elasticsearch

or

b) On the filesystem

Regarding the .yml from our docs reference yes sorry I think that is a bug / or maybe ~ means fill in the values ... yes kinda weird maybe a doc error or different convention ... I will check

I fixed the code above so it does not confuse anyone else, I should have check the reference first.

but here is the reference.... below which you can always look at that....

Please refer to the reference .yml

filebeat.reference.yml | Filebeat Reference [8.12] | Elastic

  # If enabled, instead of relying on the device ID and inode values when comparing files,
  # compare hashes of the given byte ranges in files. A file becomes an ingest target
  # when its size grows larger than offset+length (see below). Until then it's ignored.
  #prospector.scanner.fingerprint.enabled: false

  # If fingerprint mode is enabled, sets the offset from the beginning of the file
  # for the byte range used for computing the fingerprint value.
  #prospector.scanner.fingerprint.offset: 0

  # If fingerprint mode is enabled, sets the length of the byte range used for
  # computing the fingerprint value. Cannot be less than 64 bytes.
  #prospector.scanner.fingerprint.length: 1024

So yes it should look like

  # If enabled, instead of relying on the device ID and inode values when comparing files,
  # compare hashes of the given byte ranges in files. A file becomes an ingest target
  # when its size grows larger than offset+length (see below). Until then it's ignored.
  prospector.scanner.fingerprint.enabled: true

  # If fingerprint mode is enabled, sets the offset from the beginning of the file
  # for the byte range used for computing the fingerprint value.
  prospector.scanner.fingerprint.offset: 0

  # If fingerprint mode is enabled, sets the length of the byte range used for
  # computing the fingerprint value. Cannot be less than 64 bytes.
  prospector.scanner.fingerprint.length: 1024

[kbujold_wr]

Hi @stephenb

Yes we would like to be able to mute those logs. The logs come from elastic so a). There are generated by filebeat.

To view them on our system we use

 kubectl -n monitor logs mon-filebeat-stmsv

They can be viewed on the Discover page as well

[El-k]

Hi, @kbujold_wr
Were you able to solve this?
I'm facing the same problem... but one thing: I get also warnings about files smaller than 1024 bytes but not 0, like:

[warn] cannot create a file descriptor for an ingest target "/var/log/containers/alertmanager-0_gmp-system_config-reloader-783b41888d5054437d89933748846aab311545e17b51c36f64b.log": filesize of "/var/log/containers/alertmanager-0_gmp-system_config-reloader-783b41888d5054437d89933748846aab311545e17b51c36f64b.log" is 999 bytes, expected at least 1024 bytes for fingerprinting

[stephenb (Stephen Brown)]

I think the only way you are going to be able to mute those specific logs is with an ingest pipeline and a drop processor with the correct condition...

[kbujold_wr]

@stephenb

ingest pipeline

Do you have an example on how and where to set this up?

Ultimately these logs should probably be set to info.

[GEownt]

@kbujold_wr We had the same problem where the 0 byte logs were spamming our logs with warnings. I just replaced the log.level for these logs to info in filebeat:

filebeat.autodiscover:
 providers:
   - type: kubernetes
     node: ${NODE_NAME}
	 ...

processors:
  - replace:
      when:
        contains:
          message: "is 0 bytes, expected at least 64 bytes for fingerprinting"
      fields:
        - field: "log.level"
          pattern: "warn"
          replacement: "info"

This would be the easiest way I know to set the log.level for 0 bytes to info. U can use regexp or anything else to be more clear on which event should be set to info level but in our case we don´t have any other logs with "64 bytes for fingerprinting" so the processor contains is good for us.