Files and inode recreated

Elastic Stack Beats
1

Hello all,

We have an strange case, where files are deleted and created again, so the inodes changes:

  File: ‘/logs-prod/log.log’
  Size: 33235073        Blocks: 64920      IO Block: 1048576 regular file
Device: 2ah/42d Inode: 14727774157473967565  Links: 1
Access: 2024-04-04 10:50:21.429000000 -0400
Modify: 2024-04-04 10:50:21.429000000 -0400
Change: 2024-04-04 10:50:21.429000000 -0400
 Birth: -
  File: ‘/logs-prod/log.log’
  Size: 70778683        Blocks: 138240     IO Block: 1048576 regular file
Device: 2ah/42d Inode: 14727774157473967565  Links: 1
Access: 2024-04-04 10:51:21.749000000 -0400
Modify: 2024-04-04 10:51:21.749000000 -0400
Change: 2024-04-04 10:51:21.749000000 -0400
 Birth: -
  File: ‘/logs-prod/log.log’
  Size: 107995136       Blocks: 210928     IO Block: 1048576 regular file
Device: 2ah/42d Inode: 14727774157473967565  Links: 1
Access: 2024-04-04 10:52:22.005000000 -0400
Modify: 2024-04-04 10:52:22.005000000 -0400
Change: 2024-04-04 10:52:22.005000000 -0400
 Birth: -
  File: ‘/logs-prod/log.log’
  Size: 154500582       Blocks: 301760     IO Block: 1048576 regular file
Device: 2ah/42d Inode: 14727774157473967565  Links: 1
Access: 2024-04-04 10:53:27.302000000 -0400
Modify: 2024-04-04 10:53:27.302000000 -0400
Change: 2024-04-04 10:53:27.302000000 -0400
 Birth: -
  File: ‘/logs-prod/log.log’
  Size: 189079723       Blocks: 369304     IO Block: 1048576 regular file
Device: 2ah/42d Inode: 14727774157473967565  Links: 1
Access: 2024-04-04 10:54:22.500000000 -0400
Modify: 2024-04-04 10:54:22.500000000 -0400
Change: 2024-04-04 10:54:22.500000000 -0400
 Birth: -
  File: ‘/logs-prod/log.log’
  Size: 18041365        Blocks: 35240      IO Block: 1048576 regular file
Device: 2ah/42d Inode: 3240669014096275375  Links: 1
Access: 2024-04-04 10:55:17.776000000 -0400
Modify: 2024-04-04 10:55:17.776000000 -0400
Change: 2024-04-04 10:55:17.776000000 -0400
 Birth: -
  File: ‘/logs-prod/log.log’
  Size: 72428999        Blocks: 141464     IO Block: 1048576 regular file

So we configured filebeat with this parameters:

- type: log
  file_identity.path: ~
  close_timeout: 2m
  id: logs

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
   - ‘/logs-prod/log.log’

But still I can see errors with file deletion:

{"log.level":"error","@timestamp":"2024-04-04T10:32:32.294-0400","log.logger":"input.harvester","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/log.(*Log).checkFileDisappearedErrors","file.name":"log/log.go","file.line":173},"message":"Unexpected error reading from /logs-prod/log.log; error: stat /logs-prod/log.log: stale NFS file handle","service.name":"filebeat","input_id":"bf7df23f-cdf9-4475-915c-7320b030a80f","source_file":"/logs-prod/log.log","state_id":"path::/logs-prod/log.log","finished":false,"os_id":"4389460513228081789-42","old_source":"/logs-prod/log.log","old_finished":true,"old_os_id":"4389460513228081789-42","harvester_id":"d69f2cda-d00e-445f-9a06-5548a91356ee","ecs.version":"1.6.0"}

What I'm doing wrong?

Greetings,

2

This is kinda expected as reading from network volumes is not recommended and has some issues, there is not much that can be done.

This is documented here and here.

Is this file being recreated/rotated?

3

Hello Leandro,

The file is rotated:

-rw-r----- 1 1024 logs 193M 2024-04-05 03:25:40.348000000 -0400 /logs-prod/log.log
-rw-r----- 1 1024 logs 0 2024-04-05 03:25:49.403000000 -0400 /logs-prod/log.log
-rw-r----- 1 1024 logs 8.4M 2024-04-05 03:26:06.073000000 -0400 /logs-prod/log.log

-rw-r----- 1 1024 logs  20729994 Apr  5 03:20 /logs-prod/log.2024-04-05-23.log.gz
-rw-r----- 1 1024 logs  21333674 Apr  5 03:25 /logs-prod/log.2024-04-05-24.log.gz
-rw-r----- 1 1024 logs  20399755 Apr  5 03:31 /logs-prod/log.2024-04-05-25.log.gz
-rw-r----- 1 1024 logs  20951958 Apr  5 03:36 /logs-prod/log.2024-04-05-26.log.gz
-rw-r----- 1 1024 logs  21152897 Apr  5 03:41 /logs-prod/log.2024-04-05-27.log.gz
-rw-r----- 1 1024 logs  20482587 Apr  5 03:47 /logs-prod/log.2024-04-05-28.log.gz

But for example if we run the same in the server that is rotating the files, but has the same NFS. So logs are still stored in the NFS we see no issue, but if we configure it in another server that is not rotating the files we see the issue.

It's normal that filebeat is always reading the file?

COMMAND   PID             USER   FD   TYPE DEVICE  SIZE/OFF                 NODE NAME
filebeat 8349 logstash   12r   REG   0,42 196269037 11940103702398989347 /logs-prod/log.log
-rw-r----- 1 1024 logs 188M 2024-04-05 03:41:16.362000000 -0400 /logs-prod/log.log
-rw-r----- 1 1024 logs 0 2024-04-05 03:41:38.079000000 -0400 /logs-prod/log.log
COMMAND   PID             USER   FD   TYPE DEVICE SIZE/OFF                NODE NAME
filebeat 8349 logstash   21r   REG   0,42 25225753 6851282162290956458 /logs-prod/log.log
-rw-r----- 1 1024 logs 25M 2024-04-05 03:42:16.905000000 -0400 /logs-prod/log.log

Thanks for your quick answer.

4

I would say that yes, filebeat has issues with network shares, it may work, it may not work, depends on multiple things and is hard to troubleshoot.

It is not recommended to read from network shares as mentioned in the linked documentations, there are some configurations that you can try to set to improve it, but it is not guaranteed that it will work all the time.

Also, the log input is deprecated, you should try to use the filestream input is has many improvements.

You can try to use the filestream input and set the propspector scanner to use the fingerprint method as mentioned here to see if it helps.