Problems with error "File was truncated. Begin reading file from offset 0"

We use Azure Kubernetes Service, and our HTTP logs are written by the loadbalancer (ingress) to shared Azure storage. I'd like to read these logs and import them to Elasticsearch.

I realise reading from shared storage isn't supported, but I can't change the architecture of what we already have running and would like to get this to work if possible. The log files are structured in directories as "year/month/day/service" are never moved, renamed or truncated, but are deleted once their retention period has passed.

So from reading the filebeat docs, I should be able to set "file_identity" to "path" as follows:

filebeat.inputs:
    - type: log
      enabled: true
      file_identity.path: ~
      paths:
        - ${LOG_DIR}/ingress/**

However I'm still getting the error File was truncated. Begin reading file from offset 0

This is with Filebeat 7.13.4: Build info {"system_info": {"build": {"commit": "1907c246c8b0d23ae4027699c44bf3fbef57f4a4", "libbeat": "7.13.4", "time": "2021-07-14T18:42:41.000Z", "version": "7.13.4"}}}

Any suggestions? Should this work like I'm hoping?!

Hi!

Do you have any rotation taking place? If so please have a look into Log rotation results in lost or duplicate events | Filebeat Reference [7.13] | Elastic

Hi no, we don't move, rotate or rename any log files. They are in a directory structure of "/year/month/date/service" e.g. "/2021/07/29/foo.log" and then each date directory is just deleted after 60days.

If it helps, this is an example entry from the log.json file:

{"k":"filebeat::logs::path::/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","v":{"identifier_name":"path","id":"path::/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","source":"/mnt/azure-log-storage/ingress/2021/07/28/argocd-server","offset":422,"type":"log","prev_id":"","timestamp":[918211135,1627568978],"ttl":-1,"FileStateOS":{"inode":13242542234189955072,"device":76}}}

I've upped Filebeat logging to debug, and also have the following:

2021-07-29T17:26:53.095Z	DEBUG	[harvester]	log/log.go:143	File was truncated as offset (843915) > size (843466): /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n
2021-07-29T17:26:53.095Z	INFO	log/harvester.go:321	File was truncated. Begin reading file from offset 0: /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n
2021-07-29T17:26:53.095Z	DEBUG	[harvester]	log/harvester.go:608	Stopping harvester for file: /mnt/azure-log-storage/ingress/2021/07/29/oauth2-proxy_ingress-nginx-controller-7d89d7855f-p7q7n

I'm not clear why it thinks the filesize has shrunk, when I watch the filesize it appears to only increment.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.