# Problems with error "File was truncated. Begin reading file from offset 0"

**URL:** <https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 29, 2021, 9:38am UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959 "2021-07-29T09:38:14Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![mark\_vr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_vr/32/86528_2.png) [@mark\_vr](https://discuss.elastic.co/u/mark_vr)\
**Post date:** [July 29, 2021, 9:38am UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959/1 "2021-07-29T09:38:14Z")

</div>

We use Azure Kubernetes Service, and our HTTP logs are written by the loadbalancer (ingress) to shared Azure storage. I'd like to read these logs and import them to Elasticsearch.

I realise reading from shared storage isn't supported, but I can't change the architecture of what we already have running and would like to get this to work if possible. The log files are structured in directories as "year/month/day/service" are never moved, renamed or truncated, but are deleted once their retention period has passed.

So from reading the filebeat docs, I should be able to set "file\_identity" to "path" as follows:

```auto
filebeat.inputs:
    - type: log
      enabled: true
      file_identity.path: ~
      paths:
        - ${LOG_DIR}/ingress/**

```

However I'm still getting the error `File was truncated. Begin reading file from offset 0`

This is with Filebeat 7.13.4: `Build info	{"system_info": {"build": {"commit": "1907c246c8b0d23ae4027699c44bf3fbef57f4a4", "libbeat": "7.13.4", "time": "2021-07-14T18:42:41.000Z", "version": "7.13.4"}}}`

Any suggestions? Should this work like I'm hoping?!

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [July 29, 2021, 11:37am UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959/2 "2021-07-29T11:37:29Z")

</div>

Hi!

Do you have any rotation taking place? If so please have a look into [Log rotation results in lost or duplicate events | Filebeat Reference [7.13] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/file-log-rotation.html)

---

<div class="post-metadata">

**Author:** ![mark\_vr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_vr/32/86528_2.png) [@mark\_vr](https://discuss.elastic.co/u/mark_vr)\
**Post date:** [July 29, 2021, 2:18pm UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959/3 "2021-07-29T14:18:32Z")

</div>

Hi no, we don't move, rotate or rename any log files. They are in a directory structure of "/year/month/date/service" e.g. "/2021/07/29/foo.log" and then each date directory is just deleted after 60days.

---

<div class="post-metadata">

**Author:** ![mark\_vr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_vr/32/86528_2.png) [@mark\_vr](https://discuss.elastic.co/u/mark_vr)\
**Post date:** [July 29, 2021, 2:32pm UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959/4 "2021-07-29T14:32:32Z")

</div>

If it helps, this is an example entry from the `log.json` file:

```auto
{"k":"filebeat::logs::path::/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","v":{"identifier_name":"path","id":"path::/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","source":"/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","offset":422,"type":"log","prev_id":"","timestamp":[918211135,1627568978],"ttl":-1,"FileStateOS":{"inode":13242542234189955072,"device":76}}}

```

---

<div class="post-metadata">

**Author:** ![mark\_vr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_vr/32/86528_2.png) [@mark\_vr](https://discuss.elastic.co/u/mark_vr)\
**Post date:** [July 29, 2021, 5:32pm UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959/5 "2021-07-29T17:32:17Z")

</div>

I've upped Filebeat logging to debug, and also have the following:

```auto
2021-07-29T17:26:53.095Z	DEBUG	[harvester]	log/log.go:143	File was truncated as offset (843915) > size (843466): /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n
2021-07-29T17:26:53.095Z	INFO	log/harvester.go:321	File was truncated. Begin reading file from offset 0: /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n
2021-07-29T17:26:53.095Z	DEBUG	[harvester]	log/harvester.go:608	Stopping harvester for file: /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n

```

I'm not clear why it thinks the filesize has shrunk, when I watch the filesize it appears to only increment.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 26, 2021, 7:32pm UTC](https://discuss.elastic.co/t/problems-with-error-file-was-truncated-begin-reading-file-from-offset-0/279959/6 "2021-08-26T19:32:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.