# Filebeats creating duplicates on SMB logfile sources **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [November 19, 2024, 10:00pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800 "2024-11-19T22:00:57Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [November 19, 2024, 10:00pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/1 "2024-11-19T22:00:57Z")
Hi, I am faced with this challenge. The logfiles I want to watch with filebeats and then send to graylog are accessible via a SMB mount only. I already mounted with cache=none,actimeo=0 What I am seeing is that when the logfile gets a new entry at the end of the file, filebeats does notice however it rereads lots of the file and is sending out events for the past events (to the best of my knowledge the entire file). I suspect this is a problem with the logfiles being on an SMB share (from Filebeats point of view). This seems consistent with this: ```auto * log.level= info @timestamp= 2024-11-19T22:52:22.631+0100 log.logger= input.harvester log.origin.file.line= 330 log.origin.file.name= log/harvester.go log.origin.function= github.com/elastic/beats/v7/filebeat/input/log.(*Harvester).Run message= File was truncated. Begin reading file from offset 0. service.name= filebeat input_id= 0dc3d135-3a2f-4e5d-a640-397a209b52c8 source_file= /logs/c4/debug/chowmain_event_logger.log state_id= native::786563-3145860 finished= false os_id= 786563-3145860 harvester_id= 663d3ac0-04f8-4eaf-a4cf-252ae652ba0a ecs.version= **1.6.0** ``` Any ideas on how to tackle this? I have no control over the SMB server unfortunately. There currently is no log rotation in place (or not happening during the time of these tests). Whenever a new line is posted the entire file is reread and put in events. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [November 19, 2024, 10:12pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/2 "2024-11-19T22:12:06Z")
> [@j.koopmann](#): > > `File was truncated. Begin reading file from offset 0.` Can you share your filebeat config? That error is printed when Filebeat checks the size of the file and the size that we got back from the Filesystem is smaller than the offset Filebeat recorded the last time it read from the file. In other words (as an example) Filebeat thinks it's 500KB into the file but the filesystem is telling Filebeat that the file is only 400KB long. Depending on the cause, this can sometimes be addressed by enabling fingerprinting on the filestream input. [filestream input | Filebeat Reference [8.16] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-scan-fingerprint) ```auto fingerprint: enabled: true offset: 0 length: 1024 ``` ---
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [November 19, 2024, 10:13pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/3 "2024-11-19T22:13:52Z")
Sure: ```auto filebeat.inputs: - type: log enabled: true encoding: plain paths: - /logs/c4/debug/chowmain_event_logger.log fields_under_root: true fields: host.name: "ca10" ignore_older: 0 scan_frequency: 10s close_inactive: 5m close_eof: true output.console: pretty: true ``` Filebeats is running inside a docker and the mounted SMB is a volume bind in the docker. How does fingerprinting help in this case? And thanks for the fast response! ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [November 19, 2024, 10:18pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/4 "2024-11-19T22:18:09Z")
The first change you should make is to switch to the filestream input as the log input is deprecated: [filestream input | Filebeat Reference [8.16] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#filebeat-input-filestream-scan-fingerprint) I'm not entirely convinced fingerprinting will fix it but with network volumes there's no guarantee that the file ID remains stable. Fingerprinting generates the files ID using the first 1024 bytes of the file. Fingerprinting is almost always the first recommendation when network volumes are in play. The filestream input also logs the offset and the received file size from the filesystem during truncation which might be useful additional information here ---
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [November 19, 2024, 10:21pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/5 "2024-11-19T22:21:18Z")
Ignore my "how would it help". Just read the docs on this and I had mistaken it for another fingerprinting (for fields). Preliminary first test: This seems to have done the trick. It needs 1024 byte of new data/change but that is acceptable. I will monitor this. You are a lifesaver and I hate myself once more for investigating hours instead of asking the knowledgeable people earlier! Thanks. Much appreciated! ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [November 19, 2024, 10:24pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/6 "2024-11-19T22:24:15Z")
Fingers crossed that it solves your problem! ---
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [November 19, 2024, 10:27pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/7 "2024-11-19T22:27:44Z")
I also followed your advice and switch to file stream: ```auto filebeat.inputs: - type: filestream enabled: true encoding: plain paths: - /logs/c4/debug/chowmain_event_logger.log fields_under_root: true fields: host.name: "ca10" ignore_older: 0 scan_frequency: 10s close_inactive: 5m close_eof: true prospector.scanner.check_inode: false # Optional for SMB reliability ``` ---
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [December 8, 2024, 11:02pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/8 "2024-12-08T23:02:52Z")
Hi, just noticed I have duplicates again (or rather still?). I am reading every 10 seconds and every now and then I am still getting a duplicate. The message Is in the source file just once. At least not the entire file seems to be reinfected but the last lines of the log sometimes are. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [December 8, 2024, 11:41pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/9 "2024-12-08T23:41:18Z")
Are you writing directly to elasticsearch or to something else first? Can you share a log from Filebeat, the line from the file and the contents of the duplicate events from Elasticsearch? ---
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [December 9, 2024, 6:40am UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/10 "2024-12-09T06:40:43Z")
Hi, I am writing to graylog. ```auto filebeat.inputs: - type: filestream enabled: true encoding: plain paths: - /logs/c4/debug/*.log - /logs/c4/logs/*.log - /logs/c4/upmanlogs/*.log fields_under_root: true fields: host.name: "ca10" ignore_older: 0 scan_frequency: 10s close_inactive: 5m close_eof: true prospector.scanner.check_inode: false # Optional for SMB reliability fingerprint: enabled: true offset: 0 length: 1024 output.logstash: hosts: - 10.10.1.1:5044 loadbalance: true ``` Looking for a suitable example to send. Currently it is not happening. The only difference between yesterday and today is that the logfile was rotated and now is smaller. Will try to find something later to day when it becomes reproducible again. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [December 9, 2024, 6:42pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/11 "2024-12-09T18:42:22Z")
It looks like you're first writing to logstash. There are various situations with Logstash which might result in duplicate events to Elasticsearch. Both Filebeat and Logstash have "at least once" delivery guarantees which means logs may be delivered multiple times. If you are not setting the \_id of the message or otherwise fingerprinting the log, this could be expected behavior if for example the Logstash pipelines crashes and restarts or the Filebeat crashes between when it reads the log line and when it sends it to Logstash. ---
**Author:** ![j.koopmann](https://avatars.discourse-cdn.com/v4/letter/j/85e7bf/32.png) [@j.koopmann](https://discuss.elastic.co/u/j.koopmann)\ **Post date:** [December 11, 2024, 4:16pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/12 "2024-12-11T16:16:59Z")
Setting the id does not work does it? At least I was not able to set the \_id in graylog since this is a protected field. I thought that the fingerprint statement would do that exact fingerprinting? Do you have any concrete suggestion on how and esp. where to do additional fingerprinting in order to circumvent this? Currently it is not happening anymore (for no apparent reason). The components were not crashing.... ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [December 11, 2024, 4:19pm UTC](https://discuss.elastic.co/t/filebeats-creating-duplicates-on-smb-logfile-sources/370800/13 "2024-12-11T16:19:26Z")
This particular forum is for support with elastic products -- for questions about deduplicating logs sent to graylog I would recommend reaching out on the graylog community [https://community.graylog.org/](https://community.graylog.org/)