Advice on howto best run filebeat once per log file collections

stefws · March 24, 2024, 11:32am

I'm planning to daily ingest thousand of Windows client log file sets from a single linux instance of filebeat.

Our Citrix Users running a Java Client Application are logging to a Personal Windows Share mapped into their Citrix Sessions.

From a Linux VM, I've got access to all Citrix lusers log files spread across two large SMB Shares. From these two SMB shares, I every night copy desired log file collections to a local directory. Thus per date I get a local 'date' directory, from where I'm planned to run filebeat with '-once' to ingest any log files for the 'date' by generating a per run filebeat.yml pointing only the 'date' log file collection.

But question is how to configure each filebeat per ingest run, should/could every filebeat run share common directories for config, log and data (thus share registry file) ?

Thinking like this:

filebeat -c YYYY-MM-DD.runing --path.config <config dir> --path.data <data dir> --path.home /usr/share/filebeat --path.logs <log dir> -once

Having every run only use an unique filebeat.yml simply pointing to a local 'date' log file collection.

But the registry should properly not be shared I feel, in general the registry is disposable after the run once is over.

Hope my use case is understandable, otherwise ask for more info!

stefws · March 24, 2024, 12:53pm

If I launch a filebeat instance with an unique data directory, ie. an empty registry directory, with an single input section like:

- type: filestream
  id: epj-client
  enabled: true
  pipeline: pjp-epj-client-logs
  fields.index: epj-client-logs
  paths:
    - /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PER*/EPJ/*
  tags:
    - epj-client-log

having these log files here under:

$ find /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PER* -type f -exec ls -l {} \;
-rwxr-xr-x. 1 epjlog epjlog 32725 Mar 23 07:22 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERGUN/EPJ/PERGUN.Log
-rwxr-xr-x. 1 epjlog epjlog 682 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERGUN/EPJ/EPJ-log.txt
-rwxr-xr-x. 1 epjlog epjlog 32725 Mar 23 07:03 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERIHL/EPJ/PERIHL.Log
-rwxr-xr-x. 1 epjlog epjlog 472 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERIHL/EPJ/EPJ-log.txt
-rwxr-xr-x. 1 epjlog epjlog 228807 Mar 23 21:05 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNEK/EPJ/PERNEK.Log
-rwxr-xr-x. 1 epjlog epjlog 9334 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNEK/EPJ/EPJ-general-log.txt
-rwxr-xr-x. 1 epjlog epjlog 2931 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNEK/EPJ/EPJ-log.txt
-rw-r--r--. 1 epjlog epjlog 42484 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNEK/EPJ/columnajxbrowser.log
-rwxr-xr-x. 1 epjlog epjlog 164040 Mar 23 14:20 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNEN/EPJ/PERNEN.Log
-rwxr-xr-x. 1 epjlog epjlog 2530 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNEN/EPJ/EPJ-log.txt
-rwxr-xr-x. 1 epjlog epjlog 65067 Mar 23 06:36 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNYG/EPJ/PERNYG.Log
-rwxr-xr-x. 1 epjlog epjlog 879 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERNYG/EPJ/EPJ-log.txt
-rw-r--r--. 1 epjlog epjlog 16856 Mar 24 01:09 /opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERRUG/EPJ/columnajxbrowser.log

filebeat creates as expected a new registry, but only seems to register one log file from the found collection above like this:

$ cat filebeat/data/2024-03-23/registry/filebeat/log.json
{"op":"set","id":1}
{"k":"filestream::epj-client::native::412763521-64770","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERRUG/EPJ/columnajxbrowser.log","identifier_name":"native"}}}
{"op":"set","id":2}
{"k":"filestream::epj-client::native::412763521-64770","v":{"updated":[258005625852,1711282390],"cursor":null,"meta":{"identifier_name":"native","source":"/opt/epjclients/ingest/cache/2024-03-23/DC1/P/PERRUG/EPJ/columnajxbrowser.log"},"ttl":1800000000000}}

and it does not ingest any events at all, wondering why, hints appreciated!

log file of such a launch/run says:

{"log.level":"info","@timestamp":"2024-03-24T13:13:10.293+0100","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [/usr/share/pjp/filebeat] Config path: [/home/epjlog/filebeat] Data path: [/home/epjlog/filebeat/data/2024-03-23] Logs path: [/home/epjlog/filebeat/log]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.293+0100","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: d64a1af3-2d78-4c5d-a105-5fb1e4de4861","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.296+0100","log.logger":"seccomp","log.origin":{"file.name":"seccomp/seccomp.go","file.line":125},"message":"Syscall filter successfully installed","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.296+0100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"/home/epjlog/filebeat","data":"/home/epjlog/filebeat/data/2024-03-23","home":"/usr/share/filebeat","logs":"/home/epjlog/filebeat/log"},"type":"filebeat","uuid":"d64a1af3-2d78-4c5d-a105-5fb1e4de4861"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.296+0100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"480bccf4f0423099bb2c0e672a44c54ecd7a805e","libbeat":"8.10.2","time":"2023-09-18T18:09:06.000Z","version":"8.10.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.296+0100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1315},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.20.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.296+0100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1321},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2024-03-24T01:41:32+01:00","containerized":false,"name":"<redacted>","ip":["127.0.0.1","::1",<redacted>","<redacted>"],"kernel_version":"5.14.0-362.24.1.el9_3.x86_64","mac":["<redacted>"],"os":{"type":"linux","family":"redhat","platform":"rhel","name":"Red Hat Enterprise Linux","version":"9.3 (Plow)","major":9,"minor":3,"patch":0,"codename":"Plow"},"timezone":"CET","timezone_offset_sec":3600,"id":"c9b37f21204841c0a981251a75551fdc"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.297+0100","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1350},"message":"Process info","service.name":"filebeat","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","perfmon","bpf","checkpoint_restore"],"ambient":null},"cwd":"/home/epjlog","exe":"/usr/share/filebeat/bin/filebeat","name":"filebeat","pid":8002,"ppid":6857,"seccomp":{"mode":"filter","no_new_privs":true},"start_time":"2024-03-24T13:13:10.090+0100"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.297+0100","log.origin":{"file.name":"instance/beat.go","file.line":329},"message":"Setup Beat: filebeat; Version: 8.10.2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-03-24T13:13:10.299+0100","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.299+0100","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: <redacted>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-03-24T13:13:10.299+0100","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":107},"message":"SSL/TLS verifications disabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: <redacted>","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":135},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.origin":{"file.name":"instance/beat.go","file.line":515},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/home/epjlog/filebeat/data/2024-03-23/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for '/home/epjlog/filebeat/data/2024-03-23/registry/filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"input","log.origin":{"file.name":"shipper/input.go","file.line":56},"message":"creating new InputManager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":107},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.300+0100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.fields.index filebeat.inputs.0.id filebeat.inputs.0.parsers.0.multiline.match filebeat.inputs.0.parsers.0.multiline.max_lines filebeat.inputs.0.parsers.0.multiline.negate filebeat.inputs.0.parsers.0.multiline.pattern filebeat.inputs.0.parsers.0.multiline.type filebeat.inputs.0.paths.0 filebeat.inputs.0.pipeline filebeat.inputs.0.tags.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 15282076897457082015)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":121},"message":"Input 'filestream' starting","service.name":"filebeat","id":"epj-client","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.logger":"metric_registry","log.origin":{"file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"filebeat","input_type":"filestream","id":"epj-client","key":"epj-client","uuid":"c67f5566-37f1-4b92-bc4d-34ccbddb8bd6","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"cfgfile/reload.go","file.line":223},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"beater/filebeat.go","file.line":399},"message":"Running filebeat once. Waiting for completion ...","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"beater/filebeat.go","file.line":401},"message":"All data collection completed. Shutting down.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 1 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.origin":{"file.name":"cfgfile/reload.go","file.line":225},"message":"Dynamic config reloader stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.304+0100","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":170},"message":"Stopping input: 15282076897457082015","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.logger":"metric_registry","log.origin":{"file.name":"inputmon/input.go","file.line":70},"message":"unregistering","service.name":"filebeat","input_type":"filestream","id":"epj-client","key":"epj-client","uuid":"c67f5566-37f1-4b92-bc4d-34ccbddb8bd6","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":134},"message":"Input 'filestream' stopped (goroutine)","service.name":"filebeat","id":"epj-client","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":142},"message":"Input 'filestream' stopped (runner)","service.name":"filebeat","id":"epj-client","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.origin":{"file.name":"beater/signalwait.go","file.line":88},"message":"Continue shutdown: All enqueued events being published.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":130},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":164},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.307+0100","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":135},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.310+0100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpu":{"id":"session-7.scope","stats":{"periods":0,"throttled":{"ns":0,"periods":0}}},"memory":{"id":"session-7.scope","mem":{"usage":{"bytes":63586304}}}},"cpu":{"system":{"ticks":20,"time":{"ms":20}},"total":{"ticks":100,"time":{"ms":100},"value":100},"user":{"ticks":80,"time":{"ms":80}}},"handles":{"limit":{"hard":524288,"soft":524288},"open":10},"info":{"ephemeral_id":"eda52071-a026-4d13-bc51-f335e0dfbe2a","name":"filebeat","uptime":{"ms":75},"version":"8.10.2"},"memstats":{"gc_next":35961464,"memory_alloc":18135976,"memory_sys":53081352,"memory_total":61015584,"rss":110452736},"runtime":{"goroutines":14}},"filebeat":{"events":{"active":0,"added":0,"done":0},"harvester":{"closed":1,"open_files":0,"running":0,"skipped":0,"started":1},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":1,"scans":1},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":2},"load":{"1":0,"15":0,"5":0,"norm":{"1":0,"15":0,"5":0}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.310+0100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":196},"message":"Uptime: 78.583737ms","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.310+0100","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":163},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-03-24T13:13:10.310+0100","log.origin":{"file.name":"instance/beat.go","file.line":527},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}

Wondering about:

why the harvester does not pick up all log files in the 'collection'
why no events seems send to the elasticsearch output

system · April 21, 2024, 2:53pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.