Filebeats creating duplicates on SMB logfile sources

Elastic Stack Beats
1

Hi,

I am faced with this challenge. The logfiles I want to watch with filebeats and then send to graylog are accessible via a SMB mount only. I already mounted with

cache=none,actimeo=0

What I am seeing is that when the logfile gets a new entry at the end of the file, filebeats does notice however it rereads lots of the file and is sending out events for the past events (to the best of my knowledge the entire file). I suspect this is a problem with the logfiles being on an SMB share (from Filebeats point of view).

This seems consistent with this:

* log.level=
<b>info</b>
@timestamp=
<b>2024-11-19T22:52:22.631+0100</b>
log.logger=
<b>input.harvester</b>
log.origin.file.line=
<b>330</b>
log.origin.file.name=
<b>log/harvester.go</b>
log.origin.function=
<b>github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run</b>
message=
<b>File was truncated. Begin reading file from offset 0.</b>
service.name=
<b>filebeat</b>
input_id=
<b>0dc3d135-3a2f-4e5d-a640-397a209b52c8</b>
source_file=
<b>/logs/c4/debug/chowmain_event_logger.log</b>
state_id=
<b>native::786563-3145860</b>
finished=
<b>false</b>
os_id=
<b>786563-3145860</b>
harvester_id=
<b>663d3ac0-04f8-4eaf-a4cf-252ae652ba0a</b>
ecs.version=
**1.6.0**

Any ideas on how to tackle this? I have no control over the SMB server unfortunately. There currently is no log rotation in place (or not happening during the time of these tests). Whenever a new line is posted the entire file is reread and put in events.

2

Can you share your filebeat config?

That error is printed when Filebeat checks the size of the file and the size that we got back from the Filesystem is smaller than the offset Filebeat recorded the last time it read from the file.

In other words (as an example) Filebeat thinks it's 500KB into the file but the filesystem is telling Filebeat that the file is only 400KB long.

Depending on the cause, this can sometimes be addressed by enabling fingerprinting on the filestream input. filestream input | Filebeat Reference [8.16] | Elastic

fingerprint:
  enabled: true
  offset: 0
  length: 1024
3

Sure:

filebeat.inputs:
  - type: log
    enabled: true
    encoding: plain
    paths:
      - /logs/c4/debug/chowmain_event_logger.log
    fields_under_root: true
    fields:
      host.name: "ca10"
    ignore_older: 0
    scan_frequency: 10s
    close_inactive: 5m
    close_eof: true

output.console:
  pretty: true

Filebeats is running inside a docker and the mounted SMB is a volume bind in the docker. How does fingerprinting help in this case?

And thanks for the fast response!

4

The first change you should make is to switch to the filestream input as the log input is deprecated: filestream input | Filebeat Reference [8.16] | Elastic

I'm not entirely convinced fingerprinting will fix it but with network volumes there's no guarantee that the file ID remains stable.

Fingerprinting generates the files ID using the first 1024 bytes of the file. Fingerprinting is almost always the first recommendation when network volumes are in play.

The filestream input also logs the offset and the received file size from the filesystem during truncation which might be useful additional information here

5

Ignore my "how would it help". Just read the docs on this and I had mistaken it for another fingerprinting (for fields).

Preliminary first test: This seems to have done the trick. It needs 1024 byte of new data/change but that is acceptable. I will monitor this. You are a lifesaver and I hate myself once more for investigating hours instead of asking the knowledgeable people earlier!

Thanks. Much appreciated!

1 Like
6

Fingers crossed that it solves your problem!

7

I also followed your advice and switch to file stream:

filebeat.inputs:
  - type: filestream
    enabled: true
    encoding: plain
    paths:
      - /logs/c4/debug/chowmain_event_logger.log
    fields_under_root: true
    fields:
      host.name: "ca10"
    ignore_older: 0
    scan_frequency: 10s
    close_inactive: 5m
    close_eof: true
    prospector.scanner.check_inode: false  # Optional for SMB reliability
8

Hi,

just noticed I have duplicates again (or rather still?). I am reading every 10 seconds and every now and then I am still getting a duplicate. The message Is in the source file just once. At least not the entire file seems to be reinfected but the last lines of the log sometimes are.

9

Are you writing directly to elasticsearch or to something else first? Can you share a log from Filebeat, the line from the file and the contents of the duplicate events from Elasticsearch?

10

Hi,

I am writing to graylog.

filebeat.inputs:
  - type: filestream
    enabled: true
    encoding: plain
    paths:
      - /logs/c4/debug/*.log
      - /logs/c4/logs/*.log
      - /logs/c4/upmanlogs/*.log
    fields_under_root: true
    fields:
      host.name: "ca10"
    ignore_older: 0
    scan_frequency: 10s
    close_inactive: 5m
    close_eof: true
    prospector.scanner.check_inode: false  # Optional for SMB reliability

fingerprint:
  enabled: true
  offset: 0
  length: 1024

output.logstash:
  hosts:
    - 10.10.1.1:5044
  loadbalance: true

Looking for a suitable example to send. Currently it is not happening. The only difference between yesterday and today is that the logfile was rotated and now is smaller. Will try to find something later to day when it becomes reproducible again.

11

It looks like you're first writing to logstash. There are various situations with Logstash which might result in duplicate events to Elasticsearch.

Both Filebeat and Logstash have "at least once" delivery guarantees which means logs may be delivered multiple times. If you are not setting the _id of the message or otherwise fingerprinting the log, this could be expected behavior if for example the Logstash pipelines crashes and restarts or the Filebeat crashes between when it reads the log line and when it sends it to Logstash.

12

Setting the id does not work does it? At least I was not able to set the _id in graylog since this is a protected field. I thought that the fingerprint statement would do that exact fingerprinting? Do you have any concrete suggestion on how and esp. where to do additional fingerprinting in order to circumvent this? Currently it is not happening anymore (for no apparent reason). The components were not crashing....

13

This particular forum is for support with elastic products -- for questions about deduplicating logs sent to graylog I would recommend reaching out on the graylog community https://community.graylog.org/