Problems with error "File was truncated. Begin reading file from offset 0"

[mark_vr]

We use Azure Kubernetes Service, and our HTTP logs are written by the loadbalancer (ingress) to shared Azure storage. I'd like to read these logs and import them to Elasticsearch.

I realise reading from shared storage isn't supported, but I can't change the architecture of what we already have running and would like to get this to work if possible. The log files are structured in directories as "year/month/day/service" are never moved, renamed or truncated, but are deleted once their retention period has passed.

So from reading the filebeat docs, I should be able to set "file_identity" to "path" as follows:

filebeat.inputs:
    - type: log
      enabled: true
      file_identity.path: ~
      paths:
        - ${LOG_DIR}/ingress/**

However I'm still getting the error File was truncated. Begin reading file from offset 0

This is with Filebeat 7.13.4: Build info {"system_info": {"build": {"commit": "1907c246c8b0d23ae4027699c44bf3fbef57f4a4", "libbeat": "7.13.4", "time": "2021-07-14T18:42:41.000Z", "version": "7.13.4"}}}

Any suggestions? Should this work like I'm hoping?!

[ChrsMark (Chris Mark)]

Hi!

Do you have any rotation taking place? If so please have a look into Log rotation results in lost or duplicate events | Filebeat Reference [7.13] | Elastic

[mark_vr]

Hi no, we don't move, rotate or rename any log files. They are in a directory structure of "/year/month/date/service" e.g. "/2021/07/29/foo.log" and then each date directory is just deleted after 60days.

[mark_vr]

If it helps, this is an example entry from the log.json file:

{"k":"filebeat::logs::path::/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","v":{"identifier_name":"path","id":"path::/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","source":"/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","offset":422,"type":"log","prev_id":"","timestamp":[918211135,1627568978],"ttl":-1,"FileStateOS":{"inode":13242542234189955072,"device":76}}}

[mark_vr]

I've upped Filebeat logging to debug, and also have the following:

2021-07-29T17:26:53.095Z	DEBUG	[harvester]	log/log.go:143	File was truncated as offset (843915) > size (843466): /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n
2021-07-29T17:26:53.095Z	INFO	log/harvester.go:321	File was truncated. Begin reading file from offset 0: /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n
2021-07-29T17:26:53.095Z	DEBUG	[harvester]	log/harvester.go:608	Stopping harvester for file: /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n

I'm not clear why it thinks the filesize has shrunk, when I watch the filesize it appears to only increment.