# Filebeat reading entire mounted azure file from start

**URL:** <https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 27, 2021, 2:09pm UTC](https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394 "2021-04-27T14:09:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dthemg](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@dthemg](https://discuss.elastic.co/u/dthemg)\
**Post date:** [April 27, 2021, 2:09pm UTC](https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394/1 "2021-04-27T14:09:58Z")

</div>

Hi

I have an Azure file that I want to monitor using FileBeat. I am using the helm chart for this purpose, with the following `values.yaml`:

```auto
nodeSelector: { beta.kubernetes.io/os: linux }

  extraVolumes:
    - name: module-that-produces-logs
      persistentVolumeClaim:
        claimName: module-that-produces-logs

  extraVolumeMounts:
    - name: module-that-produces-logs
      mountPath: /mnt/log/module-that-produces-logs
      readOnly: true

  filebeatConfig:
    filebeat.yml: |
      output.elasticsearch:
        hosts: ["elasticsearch-master.logging.svc.cluster.local:9200"]
      filebeat.inputs:
        - type: log
          file_identity.path: ~
          enabled: true
          paths:
          - /mnt/log/*/*/*.log
          - /mnt/log/*/*.log
          - /mnt/log/*/*/*.txt
            - /mnt/log/*/*.txt

```

However, I am experiencing two issues that are causing me a headache:

1. Filebeat sends the entire .log file contents instead of just the latest changes to elasticsearch. I hoped that this would be resolved by using `file_identity.path`, but no such luck. Any advice on how to move forward would be appreciated.
2. This configuration creates two pods of filebeat, which subsequently mounts two instances of the azure file and sends two duplicates to elasticsearch. How do I reduce to just one pod? I cant seem to find this option in the filebeat helm chart `values.yaml`.

Thanks in advance,  
/David

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 28, 2021, 8:49am UTC](https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394/2 "2021-04-28T08:49:15Z")

</div>

How exactly did you change `file_identity`? You first had default `file_identity` and then run Filebeat a couple of times then changed it?

Migration from one `file_identity` setting to another one can be rocky if you have multiple entries for the same path. Based on the behaviour of Filebeat you are describing it is the case. I suggest you remove the registry file and start with a clean state with `file_identity.path`.

---

<div class="post-metadata">

**Author:** ![dthemg](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@dthemg](https://discuss.elastic.co/u/dthemg)\
**Post date:** [April 28, 2021, 11:43am UTC](https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394/3 "2021-04-28T11:43:02Z")

</div>

Hi! Thanks for answering. Yes that is pretty much what I did.

I tried to delete the registry-directory (`/usr/share/filebeat/data/registry`) on both nodes I am running Filebeat on, but I am still getting the same error unfortunately. For instance writing `A` then `B` then `C` into an empty test log file produces this output in kibana:

```auto
A <-correct
B <- correct
A <- A - B are the full log contents
B 
C <- correct
A <- A - B - C are the full log contents
B
C

```

---

<div class="post-metadata">

**Author:** ![dthemg](https://avatars.discourse-cdn.com/v4/letter/d/b19c9b/32.png) [@dthemg](https://discuss.elastic.co/u/dthemg)\
**Post date:** [April 30, 2021, 12:23pm UTC](https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394/4 "2021-04-30T12:23:57Z")

</div>

Hi again

It is _plausible_ that this issue happened due to the Azure file also being mounted on a Windows node - and that writing to a file from Windows somehow changes it leading to the whole file being recognized as new in Filebeat (like line-endings or something).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2021, 2:24pm UTC](https://discuss.elastic.co/t/filebeat-reading-entire-mounted-azure-file-from-start/271394/5 "2021-05-28T14:24:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.