On restarting filebeat, every filestream paths input is deemed to be truncated

[Dheeraj_Gupta (Dheeraj)]

Hi,

We are using filebeat to ship Zeek logs to logstash. We do this by defining a different filestream input for each type of log and in the input we define the path

filebeat.yml

path.home: /usr/share/filebeat
path.data: /var/lib/filebeat_dns
path.config: /etc/filebeat_top/filebeat_dns
path.logs: /var/log/filebeat
filebeat.registry.path: ${path.data}/registry
filebeat.shutdown_timeout: 10s

filebeat.config.inputs:
  enabled: true
  path: conf.d/*.yml
  reload.enabled: false

tags:
  - external
  
output.logstash:
  hosts:
    
    - 10.44.0.51:17002
    
  bulk_max_size:
    1024

And then individual paths in files like conf.d/abc.yml

- type: filestream
  id: captureloss
  paths:
    - /zeeklog/logs/current/capture_loss.log
  exclude_lines: ['^#' ]
  fields:
    type: bro_captureloss
  fields_under_root: true
  clean_removed: False
  close_removed: False
  clean_inactive: 3h
  ignore_older: 2h
  close.on_state_change.inactive: 30m

The filepaths are just symlinks maintained by zeek and at every hour the file gets renamed (to a different path) and a new file is created by zeek. Underlying filesystem is.

For past several months we have had the problem that whenever we restart filebeat, some/all tracked files are marked as truncated and re-read from start. This causes a lot of duplicate events. We initially modified our workflow so that all restarts are scheduled just after the hour mark (when files are relatively small so duplicates can be minimized) but have recently started investigating the issue.

What I can see by inspecting registry logs is that sometimes when filebeat detects that file has changed, it does not reset the offset and continues growing the offset past the last value. Could this be related to how inodes get reused by xfs?

E.g. consider the file smtp.log which is only about 11K long

# ls -li /zeeklog/logs/current/smtp.log 
1296539577 -rw-r--r--. 1 root root 11690 Feb 15 13:03 /zeeklog/logs/current/smtp.log

Registry entries for this file (File 41174684.json)

{"_key":"filestream::smtp::native::1296539577-64544","ttl":1800000000000,"updated":[1418184090423,1707974997],"cursor":{"offset":3162777},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}},

Log entries (File log.json in registry)

{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1418184090423,1707974997],"cursor":{"offset":3162777},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417682301492,1707982208],"cursor":{"offset":3163521},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417685718790,1707982216],"cursor":{"offset":3165128},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417686763700,1707982218],"cursor":{"offset":3165517},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417690511063,1707982226],"cursor":{"offset":3166797},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3166993},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417690894543,1707982228]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417694679417,1707982236],"cursor":{"offset":3167394},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417696225829,1707982242],"cursor":{"offset":3167585},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417698758765,1707982256],"cursor":{"offset":3167979},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417700691092,1707982264],"cursor":{"offset":3168363},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417703559996,1707982278],"cursor":{"offset":3168756},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417705524310,1707982284],"cursor":{"offset":3169650},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417707897120,1707982298],"cursor":{"offset":3169879},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3170262},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417709214339,1707982304]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417713881053,1707982316],"cursor":{"offset":3170658},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417716490692,1707982324],"cursor":{"offset":3171032},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3171263},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417719544924,1707982338]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417724711716,1707982346],"cursor":{"offset":3172356},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3172547},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417725737183,1707982348]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417727016359,1707982354],"cursor":{"offset":3172786},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417728249311,1707982368],"cursor":{"offset":3172978},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417730182164,1707982374],"cursor":{"offset":3173170},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417733006049,1707982388],"cursor":{"offset":3173362},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417734735897,1707982390],"cursor":{"offset":3173786},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417736379708,1707982404],"cursor":{"offset":3174467},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417736841114,1707982406],"cursor":{"offset":3174771},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417742391177,1707982422],"cursor":{"offset":3175544},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417743508293,1707982428],"cursor":{"offset":3175981},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3176191},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000,"updated":[1417745636175,1707982434]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3176588},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417748421923,1707982448]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417749950235,1707982462],"cursor":{"offset":3177250},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417750571082,1707982464],"cursor":{"offset":3177710}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417752653689,1707982470],"cursor":{"offset":3177946}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417754952349,1707982476],"cursor":{"offset":3178135},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3178560},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417757066508,1707982482]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417759470233,1707982506],"cursor":{"offset":3178937}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417761325804,1707982512],"cursor":{"offset":3179969},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417762752393,1707982516],"cursor":{"offset":3180441},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417765127372,1707982524],"cursor":{"offset":3181327},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417766949974,1707982530],"cursor":{"offset":3181562},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417769127711,1707982536],"cursor":{"offset":3181753},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3182326},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417770304847,1707982540]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3182517},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417771856323,1707982546]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417773547642,1707982570],"cursor":{"offset":3182704},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417775604100,1707982584],"cursor":{"offset":3183822},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417779426852,1707982602],"cursor":{"offset":3185027}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417782357532,1707982616],"cursor":{"offset":3185617},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417785637457,1707982630],"cursor":{"offset":3185809},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3186274},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417787269760,1707982636]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417788382132,1707982642],"cursor":{"offset":3186927},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417789094142,1707982644],"cursor":{"offset":3187158},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417792048381,1707982658],"cursor":{"offset":3187828},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417794313015,1707982664],"cursor":{"offset":3188040},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417795584837,1707982670],"cursor":{"offset":3188244},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417799144846,1707982696],"cursor":{"offset":3188675},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417800373799,1707982702],"cursor":{"offset":3189367},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417801736462,1707982708],"cursor":{"offset":3189830},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417805144106,1707982732],"cursor":{"offset":3190022},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417807502165,1707982746],"cursor":{"offset":3190992},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417809001967,1707982752],"cursor":{"offset":3191186},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417811493688,1707982762],"cursor":{"offset":3192040},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3192929},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417813346600,1707982768]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417816114634,1707982782],"cursor":{"offset":3193100},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417816851926,1707982784],"cursor":{"offset":3193332},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417818208028,1707982798],"cursor":{"offset":3193740},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417820472681,1707982812],"cursor":{"offset":3194132},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417823200178,1707982822],"cursor":{"offset":3195214},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417825060216,1707982828],"cursor":{"offset":3195603},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417827228328,1707982842],"cursor":{"offset":3195797},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417830223154,1707982854],"cursor":{"offset":3196378},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417830588424,1707982856],"cursor":{"offset":3196772},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000,"updated":[1417832191829,1707982864],"cursor":{"offset":3197149}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417834353743,1707982870],"cursor":{"offset":3197346},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000,"updated":[1417835240331,1707982884],"cursor":{"offset":3197577}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3197990},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417836463667,1707982890]}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417837743971,1707982896],"cursor":{"offset":3198458}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417839314793,1707982910],"cursor":{"offset":3198846},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}}
{"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417840219437,1707982924],"cursor":{"offset":3199017}}}

Despite apparent offset discrepency, the original and new files are read correctly and no events are missed. But now at this point, if I restart filebeat, it will detect smtp.log has been rotated and restarts reading the file from 0 again (but keeping offset at the high value).

Is this expected behaviour given current close_* and remove_* settings?

I can probably eliminate the problem by changing file_identity to maybe path but I am not sure how filebeat will behave when files are rotated by zeek at the hour or when zeek restarts. Will it detect truncation and restart reading at top?

[Dheeraj_Gupta (Dheeraj)]

Filebeat log entries for this inode (For last many days) - it seems same inode tracked the much heavier conn.log and dns.log files at some point.

/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-05T20:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T04:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T05:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T09:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T11:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T12:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T13:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/capture_loss.log'","service.name":"filebeat","id":"captureloss","source_file":"filestream::captureloss::native::1296539577-64544","path":"/zeeklog/logs/current/capture_loss.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T14:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T15:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T00:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T04:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T05:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T14:00:13.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T15:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-08T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-08T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T05:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T06:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T07:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T08:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T08:00:23.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T09:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T18:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T19:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T22:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T00:02:53.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/capture_loss.log","service.name":"filebeat","id":"captureloss","source_file":"filestream::captureloss::native::1296539577-64544","path":"/zeeklog/logs/current/capture_loss.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/capture_loss.log'","service.name":"filebeat","id":"captureloss","source_file":"filestream::captureloss::native::1296539577-64544","path":"/zeeklog/logs/current/capture_loss.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T07:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T08:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T13:00:03.653+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T14:00:03.654+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T15:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T16:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T17:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T20:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T21:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T22:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T00:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T06:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T08:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T09:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T16:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T17:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T20:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T21:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T22:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T02:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T08:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T14:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T15:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T17:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T18:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T19:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T20:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T04:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T05:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T06:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T08:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T13:00:13.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T13:50:47.930+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-13T11:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-13T12:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-13T19:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-13T20:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-13T23:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T00:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T08:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T18:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240215-1.ndjson:{"log.level":"info","@timestamp":"2024-02-15T10:51:18.342+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240215-1.ndjson:{"log.level":"info","@timestamp":"2024-02-15T11:00:03.846+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240215-1.ndjson:{"log.level":"info","@timestamp":"2024-02-15T13:00:08.342+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240215.ndjson:{"log.level":"info","@timestamp":"2024-02-15T10:17:27.004+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}
/var/log/filebeat/filebeat_instance_dns.log-20240215.ndjson:{"log.level":"info","@timestamp":"2024-02-15T10:51:10.929+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"}

[Dheeraj_Gupta (Dheeraj)]

I reproduced this on a test setup

/etc/filebeat/filebeat.yml

path.home: /usr/share/filebeat
path.data: /var/lib/filebeat
path.config: /etc/filebeat/
path.logs: /var/log/filebeat
filebeat.registry.path: ${path.data}/registry
filebeat.shutdown_timeout: 10s

filebeat.config.inputs:
  enabled: true
  path: conf.d/*.yml
  reload.enabled: false

output.file:
  enabled: true
  path: "/home/admin/filebeat_test"
  filename: testme_out.txt

logging.level: info
logging.to_files: true
logging.to_syslog: false

/etc/filebeat/conf.d/test.yml

- type: filestream
  id: test
  paths:
    - /home/admin/filebeat_test/testme.txt
  exclude_lines: ['^#' ]
  clean_removed: false
  close.on_state_change.removed: false
  clean_inactive: 3h
  ignore_older: 2h
  close.on_state_change.inactive: 30m

Directory /home/admin/filebeat_test is on XFS. In this directory a shell script outputs the current timestamp every 5 seconds to testme.txt.

I start filebeat service run and started the shell script. After some time I delete the file testme.txt while shell script is still running so that this file gets recreated when 5 seconds elapse. After 4-5 tries, I managed to get the inode number reused. So I immediately stopped the shell script (freezing the testme.txt file).

$ ll -i testme.txt
34641370 -rw-rw-r--. 1 admin admin    44 Feb 16 16:56 testme.txt

The log.json in registry reads something like this

{"op":"set","id":43}
{"k":"filestream::test::native::34641370-64770","v":{"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"},"ttl":1800000000000,"updated":[1418300815230,1708082782],"cursor":{"offset":110}}}
{"op":"set","id":44}
{"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418304197927,1708082790],"cursor":{"offset":132},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}}
{"op":"set","id":45}
{"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418304197927,1708082790],"cursor":{"offset":132},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}}
{"op":"set","id":46}
{"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418325496838,1708082856],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}}
{"op":"set","id":47}
{"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418325496838,1708082856],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}}
{"op":"set","id":48}
{"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1417395478512,1708083081],"cursor":{"offset":220},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}}

Which shows that the offset is greater than current file size.

Now if I restart filebeat service (with input file frozen), it detects that the file was truncated and re-reads the entire file. AT each run the offset of the file in registry increases too. Filebeat log reads

{"log.level":"info","@timestamp":"2024-02-16T17:11:35.107+0530","log.logger":"metric_registry","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry","file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"filebeat","input_type":"filestream","id":"test","key":"test","uuid":"4e90d7d5-a72b-4892-b737-8712ca24996b","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-02-16T17:11:35.108+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/home/admin/filebeat_test/testme.txt","service.name":"filebeat","id":"test","source_file":"filestream::test::native::34641370-64770","path":"/home/admin/filebeat_test/testme.txt","state-id":"native::34641370-64770","ecs.version":"1.6.0"}

The same lines are written to new output file as old output file

# cat /home/admin/filebeat_test/testme_out.txt-20240216-4.ndjson | jq ".message"
"1708082775"
"1708082780"
"1708082785"
"1708082790"

# cat /home/admin/filebeat_test/testme_out.txt-20240216-5.ndjson | jq ".message"
"1708082775"
"1708082780"
"1708082785"
"1708082790"

Meawhile registry log has this

{"op":"set","id":49}
{"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1417395478512,1708083081],"cursor":{"offset":220},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}}
{"op":"set","id":50}
{"k":"filestream::test::native::34641370-64770","v":{"updated":[1417448508008,1708083695],"cursor":{"offset":264},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"},"ttl":1800000000000}}

Which means despite marking the file as truncated, the registry offset was increased from 220 to 264 (exactly 44 bytes which is equal to file size). Since file size is less than offset, filebeat treats this file as truncated every time and reprocesses old entries without changing the offset.

Shouldn't registry offset be reduced to 0 when truncation is detected?

I will try by changing file_identity to path too.

[Dheeraj_Gupta (Dheeraj)]

Changing file_identity to path makes this very easy to reproduce.

I just add the following to conf.d/test.yml

file_identity.path: ~

Start the shell script (to create input file) and filebeat and let both run. After some time remove the input file, shell script will recreate the file but its size will be smaller than currently recorded offset. Now stop the shell script to freeze the file.

Every restart of filebeat will now re-read the file from start and increase its offset in log.json.

File state

$ ll 
-rw-rw-r--. 1 admin admin    22 Feb 16 17:39 testme.txt

log.json across restarts

{"op":"set","id":15}
{"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"updated":[1418068138204,1708085344],"cursor":{"offset":154},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000}}
{"op":"set","id":16}
{"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"updated":[1417637798436,1708085455],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000}}
{"op":"set","id":17}
{"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"updated":[1417637798436,1708085455],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000}}
{"op":"set","id":18}
{"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000,"updated":[1418095727288,1708085599],"cursor":{"offset":198}}}

Log entries regarding truncation across restarts

{"log.level":"info","@timestamp":"2024-02-16T17:40:55.298+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/home/admin/filebeat_test/testme.txt","service.name":"filebeat","id":"test","source_file":"filestream::test::path::/home/admin/filebeat_test/testme.txt","path":"/home/admin/filebeat_test/testme.txt","state-id":"path::/home/admin/filebeat_test/testme.txt","ecs.version":"1.6.0"}

{"log.level":"info","@timestamp":"2024-02-16T17:43:19.756+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/home/admin/filebeat_test/testme.txt","service.name":"filebeat","id":"test","source_file":"filestream::test::path::/home/admin/filebeat_test/testme.txt","path":"/home/admin/filebeat_test/testme.txt","state-id":"path::/home/admin/filebeat_test/testme.txt","ecs.version":"1.6.0"}

Outputs

# cat /home/admin/filebeat_test/testme_out.txt-20240216-1.ndjson | jq ".message"
"1708085352"
"1708085357"

# cat /home/admin/filebeat_test/testme_out.txt-20240216-2.ndjson | jq ".message"
"1708085352"
"1708085357"

Is this a quirk of clean_ and close.* settings or is this a bug?

[TiagoQueiroz]

Hi @Dheeraj_Gupta, thanks for such a detailed report!

I can see two issues here:

1. As you mentioned, inodes re being re-used by xfs, which will make Filebeat re-ingest the files. To overcome this, you will need to use a different file_identity. In your case fingerprint seems to be the best option.
2. There seems to be a bug on how the offset is persisted/updated when a file truncation happens. I'll further investigate that.

You seem to be facing both at the same time, which only aggravate the situation.

Because the file truncation Filebeat is detecting is related to the inode re-use problem, changing the file identity will likely solve all the problems you reported here.

[Dheeraj_Gupta (Dheeraj)]

Thanks for looking at this @TiagoQueiroz

I feel this a filestream bug which is easy to run into if underlying filesystem is not EXT. I found this comment by you on an older (similar thread).

[TiagoQueiroz]

I confirm I can reproduce that and indeed there is a bug in the way Filebeat is handling file truncation. It does not happen all the time and in your case @Dheeraj_Gupta it is triggered by identity reuse (either inode reuse or using file identity path as you described).

I'll post more details about my finding in the GitHub issue [Filebeat] filestream input resends whole log files after restart · Issue #36541 · elastic/beats · GitHub.