# On restarting filebeat, every filestream paths input is deemed to be truncated **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 15, 2024, 8:00am UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337 "2024-02-15T08:00:43Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 15, 2024, 8:00am UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/1 "2024-02-15T08:00:43Z")
Hi, We are using filebeat to ship Zeek logs to logstash. We do this by defining a different filestream input for each type of log and in the input we define the path `filebeat.yml` ```auto path.home: /usr/share/filebeat path.data: /var/lib/filebeat_dns path.config: /etc/filebeat_top/filebeat_dns path.logs: /var/log/filebeat filebeat.registry.path: ${path.data}/registry filebeat.shutdown_timeout: 10s filebeat.config.inputs: enabled: true path: conf.d/*.yml reload.enabled: false tags: - external output.logstash: hosts: - 10.44.0.51:17002 bulk_max_size: 1024 ``` And then individual paths in files like `conf.d/abc.yml` ```auto - type: filestream id: captureloss paths: - /zeeklog/logs/current/capture_loss.log exclude_lines: ['^#'] fields: type: bro_captureloss fields_under_root: true clean_removed: False close_removed: False clean_inactive: 3h ignore_older: 2h close.on_state_change.inactive: 30m ``` The filepaths are just symlinks maintained by zeek and at **every hour** the file gets renamed (to a different path) and a new file is created by zeek. Underlying filesystem is. For past several months we have had the problem that whenever we restart filebeat, some/all tracked files are marked as truncated and re-read from start. This causes a lot of duplicate events. We initially modified our workflow so that all restarts are scheduled just after the hour mark (when files are relatively small so duplicates can be minimized) but have recently started investigating the issue. What I can see by inspecting registry logs is that sometimes when filebeat detects that file has changed, it does not reset the offset and continues growing the offset past the last value. Could this be related to how inodes get reused by xfs? E.g. consider the file `smtp.log` which is only about 11K long ```auto # ls -li /zeeklog/logs/current/smtp.log 1296539577 -rw-r--r--. 1 root root 11690 Feb 15 13:03 /zeeklog/logs/current/smtp.log ``` Registry entries for this file (File `41174684.json`) ```auto {"_key":"filestream::smtp::native::1296539577-64544","ttl":1800000000000,"updated":[1418184090423,1707974997],"cursor":{"offset":3162777},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}, ``` Log entries (File `log.json` in registry) ```auto {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1418184090423,1707974997],"cursor":{"offset":3162777},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417682301492,1707982208],"cursor":{"offset":3163521},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417685718790,1707982216],"cursor":{"offset":3165128},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417686763700,1707982218],"cursor":{"offset":3165517},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417690511063,1707982226],"cursor":{"offset":3166797},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3166993},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417690894543,1707982228]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417694679417,1707982236],"cursor":{"offset":3167394},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417696225829,1707982242],"cursor":{"offset":3167585},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417698758765,1707982256],"cursor":{"offset":3167979},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417700691092,1707982264],"cursor":{"offset":3168363},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417703559996,1707982278],"cursor":{"offset":3168756},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417705524310,1707982284],"cursor":{"offset":3169650},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417707897120,1707982298],"cursor":{"offset":3169879},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3170262},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417709214339,1707982304]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417713881053,1707982316],"cursor":{"offset":3170658},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417716490692,1707982324],"cursor":{"offset":3171032},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3171263},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417719544924,1707982338]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417724711716,1707982346],"cursor":{"offset":3172356},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3172547},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417725737183,1707982348]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417727016359,1707982354],"cursor":{"offset":3172786},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417728249311,1707982368],"cursor":{"offset":3172978},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417730182164,1707982374],"cursor":{"offset":3173170},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417733006049,1707982388],"cursor":{"offset":3173362},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417734735897,1707982390],"cursor":{"offset":3173786},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417736379708,1707982404],"cursor":{"offset":3174467},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417736841114,1707982406],"cursor":{"offset":3174771},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417742391177,1707982422],"cursor":{"offset":3175544},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417743508293,1707982428],"cursor":{"offset":3175981},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3176191},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000,"updated":[1417745636175,1707982434]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3176588},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417748421923,1707982448]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417749950235,1707982462],"cursor":{"offset":3177250},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417750571082,1707982464],"cursor":{"offset":3177710}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417752653689,1707982470],"cursor":{"offset":3177946}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417754952349,1707982476],"cursor":{"offset":3178135},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3178560},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417757066508,1707982482]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417759470233,1707982506],"cursor":{"offset":3178937}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417761325804,1707982512],"cursor":{"offset":3179969},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417762752393,1707982516],"cursor":{"offset":3180441},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417765127372,1707982524],"cursor":{"offset":3181327},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417766949974,1707982530],"cursor":{"offset":3181562},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417769127711,1707982536],"cursor":{"offset":3181753},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3182326},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417770304847,1707982540]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3182517},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417771856323,1707982546]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417773547642,1707982570],"cursor":{"offset":3182704},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417775604100,1707982584],"cursor":{"offset":3183822},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417779426852,1707982602],"cursor":{"offset":3185027}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417782357532,1707982616],"cursor":{"offset":3185617},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417785637457,1707982630],"cursor":{"offset":3185809},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3186274},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417787269760,1707982636]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417788382132,1707982642],"cursor":{"offset":3186927},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417789094142,1707982644],"cursor":{"offset":3187158},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417792048381,1707982658],"cursor":{"offset":3187828},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417794313015,1707982664],"cursor":{"offset":3188040},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417795584837,1707982670],"cursor":{"offset":3188244},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417799144846,1707982696],"cursor":{"offset":3188675},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417800373799,1707982702],"cursor":{"offset":3189367},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417801736462,1707982708],"cursor":{"offset":3189830},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417805144106,1707982732],"cursor":{"offset":3190022},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417807502165,1707982746],"cursor":{"offset":3190992},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417809001967,1707982752],"cursor":{"offset":3191186},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417811493688,1707982762],"cursor":{"offset":3192040},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3192929},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417813346600,1707982768]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417816114634,1707982782],"cursor":{"offset":3193100},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417816851926,1707982784],"cursor":{"offset":3193332},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417818208028,1707982798],"cursor":{"offset":3193740},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417820472681,1707982812],"cursor":{"offset":3194132},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417823200178,1707982822],"cursor":{"offset":3195214},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417825060216,1707982828],"cursor":{"offset":3195603},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417827228328,1707982842],"cursor":{"offset":3195797},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"updated":[1417830223154,1707982854],"cursor":{"offset":3196378},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417830588424,1707982856],"cursor":{"offset":3196772},"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000,"updated":[1417832191829,1707982864],"cursor":{"offset":3197149}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417834353743,1707982870],"cursor":{"offset":3197346},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"identifier_name":"native","source":"/zeeklog/logs/current/smtp.log"},"ttl":1800000000000,"updated":[1417835240331,1707982884],"cursor":{"offset":3197577}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"cursor":{"offset":3197990},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417836463667,1707982890]}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417837743971,1707982896],"cursor":{"offset":3198458}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"ttl":1800000000000,"updated":[1417839314793,1707982910],"cursor":{"offset":3198846},"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"}}} {"k":"filestream::smtp::native::1296539577-64544","v":{"meta":{"source":"/zeeklog/logs/current/smtp.log","identifier_name":"native"},"ttl":1800000000000,"updated":[1417840219437,1707982924],"cursor":{"offset":3199017}}} ``` Despite apparent offset discrepency, the original and new files are read correctly and no events are missed. But now at this point, if I restart filebeat, it will detect `smtp.log` has been rotated and restarts reading the file from 0 again (but keeping offset at the high value). Is this expected behaviour given current `close_*` and `remove_*` settings? I can probably eliminate the problem by changing `file_identity` to maybe `path` but I am not sure how filebeat will behave when files are rotated by zeek at the hour or when zeek restarts. Will it detect truncation and restart reading at top? ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 15, 2024, 8:03am UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/2 "2024-02-15T08:03:34Z")
Filebeat log entries for this inode (For last many days) - it seems same inode tracked the much heavier `conn.log` and `dns.log` files at some point. ```auto /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-05T20:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T04:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T05:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T09:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T11:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T12:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T13:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/capture_loss.log'","service.name":"filebeat","id":"captureloss","source_file":"filestream::captureloss::native::1296539577-64544","path":"/zeeklog/logs/current/capture_loss.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T14:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T15:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-06T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T00:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T04:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T05:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T14:00:13.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-07T15:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-08T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240205.ndjson:{"log.level":"info","@timestamp":"2024-02-08T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T05:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T06:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T07:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T08:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T08:00:23.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T09:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T18:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T19:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T22:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-08T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T00:02:53.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/capture_loss.log","service.name":"filebeat","id":"captureloss","source_file":"filestream::captureloss::native::1296539577-64544","path":"/zeeklog/logs/current/capture_loss.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/capture_loss.log'","service.name":"filebeat","id":"captureloss","source_file":"filestream::captureloss::native::1296539577-64544","path":"/zeeklog/logs/current/capture_loss.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T07:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T08:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T13:00:03.653+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T14:00:03.654+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T15:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T16:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T17:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T20:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T21:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T22:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-09T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T00:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T06:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T08:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T09:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T16:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240208.ndjson:{"log.level":"info","@timestamp":"2024-02-10T17:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T20:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T21:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T22:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-10T23:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T02:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T08:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T14:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T15:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T17:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T18:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T19:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-11T20:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T01:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T02:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T03:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T04:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T05:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T06:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T08:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T13:00:13.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-12T13:50:47.930+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-13T11:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/dns.log","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240210.ndjson:{"log.level":"info","@timestamp":"2024-02-13T12:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/dns.log'","service.name":"filebeat","id":"dns","source_file":"filestream::dns::native::1296539577-64544","path":"/zeeklog/logs/current/dns.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-13T19:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-13T20:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-13T23:00:03.648+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T00:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T07:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/conn.log","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T08:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/conn.log'","service.name":"filebeat","id":"conn","source_file":"filestream::conn::native::1296539577-64544","path":"/zeeklog/logs/current/conn.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240213.ndjson:{"log.level":"info","@timestamp":"2024-02-14T18:00:03.647+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240215-1.ndjson:{"log.level":"info","@timestamp":"2024-02-15T10:51:18.342+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240215-1.ndjson:{"log.level":"info","@timestamp":"2024-02-15T11:00:03.846+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240215-1.ndjson:{"log.level":"info","@timestamp":"2024-02-15T13:00:08.342+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240215.ndjson:{"log.level":"info","@timestamp":"2024-02-15T10:17:27.004+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/zeeklog/logs/current/smtp.log","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} /var/log/filebeat/filebeat_instance_dns.log-20240215.ndjson:{"log.level":"info","@timestamp":"2024-02-15T10:51:10.929+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).readFromSource","file.name":"filestream/input.go","file.line":336},"message":"Reader was closed. Closing. Path='/zeeklog/logs/current/smtp.log'","service.name":"filebeat","id":"smtp","source_file":"filestream::smtp::native::1296539577-64544","path":"/zeeklog/logs/current/smtp.log","state-id":"native::1296539577-64544","ecs.version":"1.6.0"} ``` ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 16, 2024, 12:10pm UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/3 "2024-02-16T12:10:37Z")
I reproduced this on a test setup `/etc/filebeat/filebeat.yml` ```auto path.home: /usr/share/filebeat path.data: /var/lib/filebeat path.config: /etc/filebeat/ path.logs: /var/log/filebeat filebeat.registry.path: ${path.data}/registry filebeat.shutdown_timeout: 10s filebeat.config.inputs: enabled: true path: conf.d/*.yml reload.enabled: false output.file: enabled: true path: "/home/admin/filebeat_test" filename: testme_out.txt logging.level: info logging.to_files: true logging.to_syslog: false ``` `/etc/filebeat/conf.d/test.yml` ```auto - type: filestream id: test paths: - /home/admin/filebeat_test/testme.txt exclude_lines: ['^#'] clean_removed: false close.on_state_change.removed: false clean_inactive: 3h ignore_older: 2h close.on_state_change.inactive: 30m ``` Directory `/home/admin/filebeat_test` is on XFS. In this directory a shell script outputs the current timestamp every 5 seconds to `testme.txt`. I start filebeat service run and started the shell script. After some time I delete the file `testme.txt` while shell script is still running so that this file gets recreated when 5 seconds elapse. After 4-5 tries, I managed to get the inode number reused. So I immediately stopped the shell script (freezing the `testme.txt` file). ```auto $ ll -i testme.txt 34641370 -rw-rw-r--. 1 admin admin 44 Feb 16 16:56 testme.txt ``` The `log.json` in registry reads something like this ```auto {"op":"set","id":43} {"k":"filestream::test::native::34641370-64770","v":{"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"},"ttl":1800000000000,"updated":[1418300815230,1708082782],"cursor":{"offset":110}}} {"op":"set","id":44} {"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418304197927,1708082790],"cursor":{"offset":132},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}} {"op":"set","id":45} {"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418304197927,1708082790],"cursor":{"offset":132},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}} {"op":"set","id":46} {"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418325496838,1708082856],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}} {"op":"set","id":47} {"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1418325496838,1708082856],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}} {"op":"set","id":48} {"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1417395478512,1708083081],"cursor":{"offset":220},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}} ``` Which shows that the offset is greater than current file size. Now if I restart filebeat service (with input file frozen), it detects that the file was truncated and re-reads the entire file. AT each run the offset of the file in registry increases too. Filebeat log reads ```auto {"log.level":"info","@timestamp":"2024-02-16T17:11:35.107+0530","log.logger":"metric_registry","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/inputmon.NewInputRegistry","file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"filebeat","input_type":"filestream","id":"test","key":"test","uuid":"4e90d7d5-a72b-4892-b737-8712ca24996b","ecs.version":"1.6.0"} {"log.level":"info","@timestamp":"2024-02-16T17:11:35.108+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/home/admin/filebeat_test/testme.txt","service.name":"filebeat","id":"test","source_file":"filestream::test::native::34641370-64770","path":"/home/admin/filebeat_test/testme.txt","state-id":"native::34641370-64770","ecs.version":"1.6.0"} ``` The same lines are written to new output file as old output file ```auto # cat /home/admin/filebeat_test/testme_out.txt-20240216-4.ndjson | jq ".message" "1708082775" "1708082780" "1708082785" "1708082790" # cat /home/admin/filebeat_test/testme_out.txt-20240216-5.ndjson | jq ".message" "1708082775" "1708082780" "1708082785" "1708082790" ``` Meawhile registry log has this ```auto {"op":"set","id":49} {"k":"filestream::test::native::34641370-64770","v":{"ttl":1800000000000,"updated":[1417395478512,1708083081],"cursor":{"offset":220},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"}}} {"op":"set","id":50} {"k":"filestream::test::native::34641370-64770","v":{"updated":[1417448508008,1708083695],"cursor":{"offset":264},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"native"},"ttl":1800000000000}} ``` Which means despite marking the file as truncated, the registry offset was increased from 220 to 264 (exactly 44 bytes which is equal to file size). Since file size is less than offset, filebeat treats this file as truncated every time and reprocesses old entries without changing the offset. Shouldn't registry offset be reduced to 0 when truncation is detected? I will try by changing `file_identity` to `path` too. ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 16, 2024, 12:39pm UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/4 "2024-02-16T12:39:45Z")
Changing `file_identity` to `path` makes this very easy to reproduce. I just add the following to `conf.d/test.yml` ```auto file_identity.path: ~ ``` Start the shell script (to create input file) and filebeat and let both run. After some time remove the input file, shell script will recreate the file but its size will be smaller than currently recorded offset. Now stop the shell script to freeze the file. Every restart of filebeat will now re-read the file from start and increase its offset in `log.json`. File state ```auto $ ll -rw-rw-r--. 1 admin admin 22 Feb 16 17:39 testme.txt ``` `log.json` across restarts ```auto {"op":"set","id":15} {"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"updated":[1418068138204,1708085344],"cursor":{"offset":154},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000}} {"op":"set","id":16} {"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"updated":[1417637798436,1708085455],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000}} {"op":"set","id":17} {"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"updated":[1417637798436,1708085455],"cursor":{"offset":176},"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000}} {"op":"set","id":18} {"k":"filestream::test::path::/home/admin/filebeat_test/testme.txt","v":{"meta":{"source":"/home/admin/filebeat_test/testme.txt","identifier_name":"path"},"ttl":1800000000000,"updated":[1418095727288,1708085599],"cursor":{"offset":198}}} ``` Log entries regarding truncation across restarts ```auto {"log.level":"info","@timestamp":"2024-02-16T17:40:55.298+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/home/admin/filebeat_test/testme.txt","service.name":"filebeat","id":"test","source_file":"filestream::test::path::/home/admin/filebeat_test/testme.txt","path":"/home/admin/filebeat_test/testme.txt","state-id":"path::/home/admin/filebeat_test/testme.txt","ecs.version":"1.6.0"} ``` ```auto {"log.level":"info","@timestamp":"2024-02-16T17:43:19.756+0530","log.logger":"input.filestream","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).openFile","file.name":"filestream/input.go","file.line":274},"message":"File was truncated. Reading file from offset 0. Path=/home/admin/filebeat_test/testme.txt","service.name":"filebeat","id":"test","source_file":"filestream::test::path::/home/admin/filebeat_test/testme.txt","path":"/home/admin/filebeat_test/testme.txt","state-id":"path::/home/admin/filebeat_test/testme.txt","ecs.version":"1.6.0"} ``` Outputs ```auto # cat /home/admin/filebeat_test/testme_out.txt-20240216-1.ndjson | jq ".message" "1708085352" "1708085357" # cat /home/admin/filebeat_test/testme_out.txt-20240216-2.ndjson | jq ".message" "1708085352" "1708085357" ``` Is this a quirk of `clean_` and `close.*` settings or is this a bug? ---
**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\ **Post date:** [February 19, 2024, 11:03am UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/5 "2024-02-19T11:03:40Z")
Hi @Dheeraj_Gupta, thanks for such a detailed report! I can see two issues here: 1. As you mentioned, inodes re being re-used by xfs, which will make Filebeat re-ingest the files. To overcome this, you will need to use a different `file_identity`. In your case [`fingerprint`](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-filestream.html#_file_identity_2) seems to be the best option. 2. There seems to be a bug on how the offset is persisted/updated when a file truncation happens. I'll further investigate that. You seem to be facing both at the same time, which only aggravate the situation. Because the file truncation Filebeat is detecting is related to the inode re-use problem, changing the file identity will likely solve all the problems you reported here. ---
**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\ **Post date:** [February 19, 2024, 11:26am UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/6 "2024-02-19T11:26:22Z")
Thanks for looking at this @TiagoQueiroz I feel this a filestream bug which is easy to run into if underlying filesystem is not EXT. I found [this comment](https://discuss.elastic.co/t/filestream-input-sends-duplicates-events-on-restart-and-during-operation/334951/19) by you on an older (similar thread). ---
**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\ **Post date:** [February 20, 2024, 1:32pm UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/7 "2024-02-20T13:32:05Z")
I confirm I can reproduce that and indeed there is a bug in the way Filebeat is handling file truncation. It does not happen all the time and in your case @Dheeraj_Gupta it is triggered by identity reuse (either inode reuse or using file identity path as you described). I'll post more details about my finding in the GitHub issue [[Filebeat] filestream input resends whole log files after restart · Issue #36541 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/36541). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 19, 2024, 3:32pm UTC](https://discuss.elastic.co/t/on-restarting-filebeat-every-filestream-paths-input-is-deemed-to-be-truncated/353337/8 "2024-03-19T15:32:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.