Why Filebeat cannot start and work correctly on windows

Elastic Stack Beats
1

I want to install filebeat on my windows server . My filebeat version is : 8.10.4
This is my filebeat config :

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://10.20.11.29:9200"]
  protocol: "https"

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "password"
#  index: audit-db
  ssl:
    enabled: true
    ca_trusted_fingerprint: "23E9D474BE99BB2C6C331F43C96YU35B3096ABA83C822BB3AB06672F95AE84BB"
output.elasticsearch.index: "veem-%{[agent.version]}"
setup.template.name: "veem"
setup.template.pattern: "veem-%{[agent.version]}"
setup.dashboards.index: "veem-*"

when I type :

 .\filebeat.exe setup -e

It is showing follow output :

Exiting: error loading template: failed to put data stream: could not put data stream: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [veem]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [veem]"},"status":400}. Response body: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"no matching index template found for data stream [veem]"}],"type":"illegal_argument_exception","reason":"no matching index template found for data stream [veem]"},"status":400}
2

Think there is a bug in the docs... same result on my mac try
This worked for me.

setup.template.name: "veem-%{[agent.version]}"
setup.template.pattern: "veem-%{[agent.version]}"
setup.dashboards.index: "veem-*"
3

That is so strange when I use Service-Start filebeat . my filebeat service will start
But it does not create any datastream and index template .but when I type :

.\filebeat.exe setup -e

it will show follow and create Datastream with the name veeam and alsi index template

{"log.level":"info","@timestamp":"2023-11-13T15:24:06.432+0330","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [C:\\filebeat] Config path: [C:\\filebeat] Data path: [C:\\filebeat\\data] Logs path: [C:\\filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.434+0330","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: d20f6473-a5e3-41b3-ae7b-1ef66b98e7ae","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.453+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\\filebeat","data":"C:\\filebeat\\data","home":"C:\\filebeat","logs":"C:\\filebeat\\logs"},"type":"filebeat","uuid":"d20f6473-a5e3-41b3-ae7b-1ef66b98e7ae"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.453+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"10b198c985eb95c16405b979c63847881a199aba","libbeat":"8.10.4","time":"2023-10-11T19:23:15.000Z","version":"8.10.4"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.457+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1315},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.20.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.463+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1321},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-11-13T13:49:36+03:30","name":"tshoot-veeam","ip":["169.254.188.88","172.20.110.240","::1","127.0.0.1"],"kernel_version":"10.0.17763.3406 (WinBuild.160101.0800)","mac":["00:50:56:ab:b0:16","00:50:56:ab:77:b4"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.3406"},"timezone":"+0330","timezone_offset_sec":12600,"id":"596997d0-9107-4b94-95fe-d12941c59cff"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.466+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1350},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\filebeat","exe":"C:\\filebeat\\filebeat.exe","name":"filebeat.exe","pid":6980,"ppid":3452,"start_time":"2023-11-13T15:24:05.911+0330"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T15:24:06.467+0330","log.origin":{"file.name":"instance/beat.go","file.line":329},"message":"Setup Beat: filebeat; Version: 8.10.4","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-11-13T15:24:07.793+0330","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.793+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://172.20.112.29:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.797+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: Tshoot-veeam","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.798+0330","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":135},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.799+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://172.20.112.29:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.822+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":179},"message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.823+0330","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls_config.go","file.line":199},"message":"CA certificate matching 'ca_trusted_fingerprint' found, adding it to 'certificate_authorities'","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.836+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":291},"message":"Attempting to connect to Elasticsearch version 8.10.4","service.name":"filebeat","ecs.version":"1.6.0"}
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

{"log.level":"info","@timestamp":"2023-11-13T15:24:07.838+0330","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":230},"message":"Auto ILM enable success.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.842+0330","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":118},"message":"ILM policy filebeat exists already.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.843+0330","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":365},"message":"Set settings.index.lifecycle.name in template to {filebeat {\"policy\":{\"phases\":{\"hot\":{\"actions\":{\"rollover\":{\"max_age\":\"30d\",\"max_primary_shard_size\":\"50gb\"}}}}}}} as ILM is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:07.849+0330","log.logger":"template","log.origin":{"file.name":"template/load.go","file.line":245},"message":"Existing template will be overwritten, as overwrite is enabled.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.093+0330","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":159},"message":"Try loading template veeam-8.10.4 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.369+0330","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":127},"message":"Template with name \"veeam-8.10.4\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.371+0330","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":184},"message":"Try loading data stream veeam-8.10.4 to Elasticsearch","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.453+0330","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.659+0330","log.logger":"template_loader","log.origin":{"file.name":"template/load.go","file.line":150},"message":"Data stream with name \"veeam-8.10.4\" loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.659+0330","log.logger":"index-management","log.origin":{"file.name":"idxmgmt/std.go","file.line":266},"message":"Loaded index template.","service.name":"filebeat","ecs.version":"1.6.0"}
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
{"log.level":"info","@timestamp":"2023-11-13T15:24:09.665+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://172.20.112.29:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T15:24:11.287+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://172.20.112.29:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-11-13T15:24:11.454+0330","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":91},"message":"error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, canceled, context deadline exceeded.","service.name":"filebeat","ecs.version":"1.6.0"}

What is the issue why it cannot create datastream and index template after run?

4

Also this is my log file when I am running filebeat service on windows :

{"log.level":"info","@timestamp":"2023-11-13T17:02:11.361+0330","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [C:\\filebeat] Config path: [C:\\filebeat] Data path: [C:\\ProgramData\\filebeat] Logs path: [C:\\ProgramData\\filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:11.364+0330","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: 26c776cb-0843-465d-9932-5d8b10f5c2f3","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-11-13T17:02:14.379+0330","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:14.383+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\\filebeat","data":"C:\\ProgramData\\filebeat","home":"C:\\filebeat","logs":"C:\\ProgramData\\filebeat\\logs"},"type":"filebeat","uuid":"26c776cb-0843-465d-9932-5d8b10f5c2f3"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T17:02:14.383+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"a8dbc6c06381f4fe33a5dc23906d63c04c9e2444","libbeat":"8.7.0","time":"2023-03-23T00:44:06.000Z","version":"8.7.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T17:02:14.383+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.19.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T17:02:14.395+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1114},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-11-13T16:54:26+03:30","name":"Tshoot-veeam","ip":["169.254.188.88","172.20.110.240","::1","127.0.0.1"],"kernel_version":"10.0.17763.3406 (WinBuild.160101.0800)","mac":["00:50:56:ab:b0:16","00:50:56:ab:77:b4"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.3406"},"timezone":"+0330","timezone_offset_sec":12600,"id":"596997d0-9107-4b94-95fe-d12941c59cff"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T17:02:14.396+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1143},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\Windows\\system32","exe":"C:\\filebeat\\filebeat.exe","name":"filebeat.exe","pid":1112,"ppid":804,"start_time":"2023-11-13T17:02:09.912+0330"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T17:02:14.396+0330","log.origin":{"file.name":"instance/beat.go","file.line":297},"message":"Setup Beat: filebeat; Version: 8.7.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-11-13T17:02:15.725+0330","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.725+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://172.20.112.29:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.725+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: Tshoot-veeam","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.725+0330","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.726+0330","log.origin":{"file.name":"instance/beat.go","file.line":486},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.726+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.728+0330","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\ProgramData\\filebeat\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.729+0330","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\ProgramData\\filebeat\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.729+0330","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.729+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.729+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.729+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 1278445034110928204)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.729+0330","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'filestream' starting","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.730+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.730+0330","log.origin":{"file.name":"cfgfile/reload.go","file.line":164},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:15.730+0330","log.origin":{"file.name":"cfgfile/reload.go","file.line":224},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:17.382+0330","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-13T17:02:45.728+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":140}},"total":{"ticks":233,"time":{"ms":233},"value":233},"user":{"ticks":93,"time":{"ms":93}}},"info":{"ephemeral_id":"c2515815-fd79-4eeb-8db2-01548d2b5a52","name":"filebeat","uptime":{"ms":34441},"version":"8.7.0"},"memstats":{"gc_next":21580128,"memory_alloc":10810272,"memory_sys":31915272,"memory_total":51676152,"rss":56111104},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":4},"handles":{"open":243}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-13T17:03:15.728+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":140},"total":{"ticks":233,"value":233},"user":{"ticks":93}},"info":{"ephemeral_id":"c2515815-fd79-4eeb-8db2-01548d2b5a52","uptime":{"ms":64442},"version":"8.7.0"},"memstats":{"gc_next":21580128,"memory_alloc":10866536,"memory_total":51732416,"rss":56094720},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}

This is so strange why it cannot create datastream / index automatically after run filebeat service although it is working excellent with same filebeat.yml file on Linux

5

@baber1223

Did you try my suggestion?

6

@stephenb

Yes . I have changed it and as I said when I am running filebeat in foreground it is creating datastream and index template

.\filebeat.exe setup -e

but when I want to run it as server it will run in the windows service but does not create any datastream and index template so not working correctly .
I try filebeat 8.7 and 8.10.4

Although this config and method is working on my Linux systems

7

So running as a filebeat as a Windows Service does not automatically run setup you need to run that before you start the service... not sure I am understanding your steps correctly but just start filebeat as a service does not load all the assets.

This is my understanding.

You do NOT need to run setup everytime just once for each time you add new modules etc...

Not to say there is not a bug... but I just ran filebeat 8.10.4 on a windows box and it performed as I expected.

I ran Setup then I ran the Service just as explained in the Filebeat Quickstart and everything worked as expected

8

@stephenb
So I did follow steps :

1- .\install-service-filebeat.ps1 ------> successfull
2- .\filebeat.exe ----- in this step it does not show any output I am still waiting

9

Please follow the guide I provided with the exact steps... not skipping them .. not executing them differently... apologies but I can not help, if you are going to ignore the docs and just execute random commands,

setup
PS > .\filebeat.exe setup -e

Just running just starts filebeat with no output...

PS > .\filebeat.exe

If you want to test filebeat in the foreground

PS > .\filebeat.exe -e

that will print the logs to the console.

I will suggest... Start Over and follow the Quick Start Guide closely.

10

@stephenb

Please see attached pic as you can see .\flebeat.exec is running more than 10 min
and also filebeat service is stop . What should I do now ?

ff
ff858×411 111 KB

11

Right ... so Filebeat is running in the foreground with No Logging because you did not put the -e in so nothing will be printed to the screen...

If you add the -e you will see the logs and we will see if there is an issue... other wise filebeat is running ... there may be no logs to ingest...

If you want to see what is happening run the command as I showed

Screenshot 2023-11-13 at 11.56.26 AM
Screenshot 2023-11-13 at 11.56.26 AM1920×920 159 KB

12

I am really confused . Does it your mean not important if filebeat as service is running or not ? Do I have to use just follow commands ? :

PS > .\filebeat.exe setup -e


PS > .\filebeat.exe -e

as I said before that when I run it in the forground it will create datastream and index templates but when I start it in the background does not create datastream and index .....

13

If you run setup first

PS > .\filebeat.exe setup -e

Then start as a Service

PS > Start-Service filebeat

It shoudl work as expected if it does not then there is another issue.

filbeat.exe is and executable

This sets up assets

  1. PS > .\filebeat.exe setup -e

This runs filebeat in the foreground NOT as a service it is good for debugging the filebeat information logs will come to the console, but the process will die when you close the window

  1. PS > .\filebeat.exe -e

This runs filebeat as a Windows Service

  1. PS > Start-Service filebeat

Do not try to run filebeat in the foreground and windows service at the same time ... i.e. pick command 2 or 3 but not both.

Commands 2 and 3 are different ways to run filebeat

Command 1 sets up the assets whether you run with 2 or 3

You never did answer did you change your filebeat.yml as I suggested?
Did you clean out the templates, datastreams?
Then run the setup command above
Then start filebeat as a service.

Why filebeat works one way for you and not another I do not know because I am having a hard time understanding your steps.

14

@stephenb

I did follow steps :

change yml file such as follow :

setup.template.name: "veem-%{[agent.version]}"
setup.template.pattern: "veem-%{[agent.version]}"
setup.dashboards.index: "veem-*"

2 -

PS > .\filebeat.exe setup -e

and is showing follow log

{"log.level":"info","@timestamp":"2023-11-14T13:05:55.058+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12721619},"version":"8.10.4"},"memstats":{"gc_next":34338112,"memory_alloc":16896168,"memory_total":585449992,"rss":70778880},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:06:25.056+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12751618},"version":"8.10.4"},"memstats":{"gc_next":34338112,"memory_alloc":16948080,"memory_total":585501904,"rss":70766592},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"handles":{"open":-2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:06:55.059+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12781620},"version":"8.10.4"},"memstats":{"gc_next":34342704,"memory_alloc":16770576,"memory_total":585551336,"rss":70778880},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:07:25.058+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12811618},"version":"8.10.4"},"memstats":{"gc_next":34342704,"memory_alloc":16831120,"memory_total":585611880,"rss":70778880},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:07:55.057+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12841618},"version":"8.10.4"},"memstats":{"gc_next":34342704,"memory_alloc":16878536,"memory_total":585659296,"rss":70778880},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:08:25.050+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12871610},"version":"8.10.4"},"memstats":{"gc_next":34342704,"memory_alloc":16925320,"memory_total":585706080,"rss":70778880},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:08:55.059+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1234},"total":{"ticks":5577,"value":5577},"user":{"ticks":4343}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","uptime":{"ms":12901620},"version":"8.10.4"},"memstats":{"gc_next":34337952,"memory_alloc":16773232,"memory_total":585754840,"rss":70778880},"runtime":{"goroutines":29}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.761+0330","log.logger":"service","log.origin":{"file.name":"service/service.go","file.line":52},"message":"Received signal \"interrupt\", stopping","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.761+0330","log.logger":"service_windows","log.origin":{"file.name":"service/service_windows.go","file.line":75},"message":"received state change 'svc.Stop' from windows service manager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.761+0330","log.origin":{"file.name":"beater/filebeat.go","file.line":484},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.763+0330","log.logger":"service_windows","log.origin":{"file.name":"service/service_windows.go","file.line":89},"message":"changed windows service state to svc.StopPending, invoking stopCallback","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.764+0330","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.764+0330","log.logger":"service","log.origin":{"file.name":"service/service.go","file.line":59},"message":"Received Windows SVC stop/shutdown request","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.764+0330","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 1 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.765+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":170},"message":"Stopping input: 1278445034110928204","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.765+0330","log.origin":{"file.name":"cfgfile/reload.go","file.line":225},"message":"Dynamic config reloader stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.765+0330","log.logger":"metric_registry","log.origin":{"file.name":"inputmon/input.go","file.line":70},"message":"unregistering","service.name":"filebeat","input_type":"filestream","id":"my-filestream-id","key":"my-filestream-id","uuid":"ed5aa150-e891-4e00-957a-e187bfda639d","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.766+0330","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":134},"message":"Input 'filestream' stopped (goroutine)","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.766+0330","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":142},"message":"Input 'filestream' stopped (runner)","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.767+0330","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.767+0330","log.origin":{"file.name":"beater/filebeat.go","file.line":484},"message":"Stopping filebeat","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.767+0330","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":130},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.767+0330","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":164},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.768+0330","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":135},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.770+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":1250,"time":{"ms":1250}},"total":{"ticks":5593,"time":{"ms":5593},"value":5593},"user":{"ticks":4343,"time":{"ms":4343}}},"info":{"ephemeral_id":"ee81fb90-7cbc-4862-bb95-5109194add55","name":"filebeat","uptime":{"ms":12919330},"version":"8.10.4"},"memstats":{"gc_next":34337952,"memory_alloc":16865000,"memory_sys":140067976,"memory_total":585846608,"rss":70848512},"runtime":{"goroutines":13}},"filebeat":{"events":{"active":0,"added":0,"done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":1,"scans":1},"output":{"batches":{"split":0},"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"elasticsearch","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"processor":{"add_host_metadata":{"fqdn_lookup_failed":0}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":10},"handles":{"open":294}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.771+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":196},"message":"Uptime: 3h35m19.3343308s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.771+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":163},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:09:12.772+0330","log.origin":{"file.name":"instance/beat.go","file.line":527},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
Loaded Ingest pipelines
PS C:\filebeat>

finally follow comand :


PS C:\filebeat> Start-Service filebeat
PS C:\filebeat>

Now as you can see on dataview veeam it cannot show anything

kb1
kb11898×911 122 KB

Also this is my output :

PS C:\filebeat> .\filebeat.exe -e

PS C:\filebeat> .\filebeat.exe -e
{"log.level":"info","@timestamp":"2023-11-14T13:56:49.869+0330","log.origin":{"file.name":"instance/beat.go","file.line":783},"message":"Home path: [C:\\filebeat] Config path: [C:\\filebeat] Data path: [C:\\filebeat\\data] Logs path: [C:\\filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:49.879+0330","log.origin":{"file.name":"instance/beat.go","file.line":791},"message":"Beat ID: da9e01b8-fe35-4a7d-8fc2-dfdc54b6895b","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:49.969+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1303},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\\filebeat","data":"C:\\filebeat\\data","home":"C:\\filebeat","logs":"C:\\filebeat\\logs"},"type":"filebeat","uuid":"da9e01b8-fe35-4a7d-8fc2-dfdc54b6895b"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:56:49.969+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1312},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"10b198c985eb95c16405b979c63847881a199aba","libbeat":"8.10.4","time":"2023-10-11T19:23:15.000Z","version":"8.10.4"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:56:49.971+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1315},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":10,"version":"go1.20.8"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:56:50.000+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1321},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-11-14T00:30:55+03:30","name":"veeam","ip":["172.20.110.24","172.20.108.254","172.20.106.30","172.20.105.253","169.254.12.242","169.254.143.161","172.20.107.125","169.254.153.187","172.16.25.254","172.20.103.126","::1","127.0.0.1"],"kernel_version":"10.0.17763.4252 (WinBuild.160101.0800)","mac":["00:50:56:a9:d2:ee","00:50:56:ab:52:34","00:50:56:ab:85:28","00:50:56:ab:8b:0c","00:50:56:ab:99:03","00:50:56:ab:dd:ca","00:50:56:ab:de:92","00:50:56:ab:e4:3e","00:50:56:ab:f0:35","00:50:56:ab:f7:ca"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2019 Datacenter","version":"10.0","major":10,"minor":0,"patch":0,"build":"17763.4252"},"timezone":"+0330","timezone_offset_sec":12600,"id":"c9c6644e-428a-4f64-a069-3707d6cb887d"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:56:50.002+0330","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1350},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\filebeat","exe":"C:\\filebeat\\filebeat.exe","name":"filebeat.exe","pid":15228,"ppid":17380,"start_time":"2023-11-14T13:56:49.306+0330"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:56:50.004+0330","log.origin":{"file.name":"instance/beat.go","file.line":329},"message":"Setup Beat: filebeat; Version: 8.10.4","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-11-14T13:56:51.382+0330","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:51.383+0330","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://172.20.112.29:9200","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:51.385+0330","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":105},"message":"Beat name: veeam","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:51.385+0330","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":135},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:51.385+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:51.386+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://172.20.112.29:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:52.975+0330","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":100},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:56:53.076+0330","log.logger":"kibana","log.origin":{"file.name":"kibana/client.go","file.line":183},"message":"Kibana url: http://172.20.112.29:5601","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-11-14T13:56:54.975+0330","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":91},"message":"error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, canceled, context deadline exceeded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:57:21.391+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":421,"time":{"ms":421}},"total":{"ticks":3014,"time":{"ms":3014},"value":3014},"user":{"ticks":2593,"time":{"ms":2593}}},"info":{"ephemeral_id":"2c2f6db7-cf63-4a86-8e3a-a4315a8352ad","name":"filebeat","uptime":{"ms":31602},"version":"8.10.4"},"memstats":{"gc_next":45901456,"memory_alloc":37925632,"memory_sys":139494536,"memory_total":512994688,"rss":101941248},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0},"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":10},"handles":{"open":282}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:57:51.399+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":500,"time":{"ms":79}},"total":{"ticks":3703,"time":{"ms":689},"value":3703},"user":{"ticks":3203,"time":{"ms":610}}},"info":{"ephemeral_id":"2c2f6db7-cf63-4a86-8e3a-a4315a8352ad","uptime":{"ms":61610},"version":"8.10.4"},"memstats":{"gc_next":37328136,"memory_alloc":21195600,"memory_total":541887920,"rss":88252416},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"handles":{"open":2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.712+0330","log.origin":{"file.name":"instance/beat.go","file.line":996},"message":"Kibana dashboards successfully loaded.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.712+0330","log.origin":{"file.name":"instance/beat.go","file.line":515},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.721+0330","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\filebeat\\data\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.723+0330","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\filebeat\\data\\registry\\filebeat'. Active transaction id=0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.723+0330","log.logger":"input","log.origin":{"file.name":"shipper/input.go","file.line":56},"message":"creating new InputManager","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.724+0330","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":107},"message":"States Loaded from registrar: 0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.725+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.725+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.726+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 1278445034110928204)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.726+0330","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":121},"message":"Input 'filestream' starting","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.727+0330","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":106},"message":"Loading and starting Inputs completed. Enabled inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.727+0330","log.origin":{"file.name":"cfgfile/reload.go","file.line":163},"message":"Config reloader started","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.727+0330","log.logger":"metric_registry","log.origin":{"file.name":"inputmon/input.go","file.line":63},"message":"registering","service.name":"filebeat","input_type":"filestream","id":"my-filestream-id","key":"my-filestream-id","uuid":"79b2f689-c3bd-453b-a702-41ac5e4c6e50","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:14.728+0330","log.origin":{"file.name":"cfgfile/reload.go","file.line":223},"message":"Loading of config files completed.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-14T13:58:21.391+0330","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":640,"time":{"ms":140}},"total":{"ticks":4108,"time":{"ms":405},"value":4108},"user":{"ticks":3468,"time":{"ms":265}}},"info":{"ephemeral_id":"2c2f6db7-cf63-4a86-8e3a-a4315a8352ad","uptime":{"ms":91602},"version":"8.10.4"},"memstats":{"gc_next":38151208,"memory_alloc":26264752,"memory_total":563093272,"rss":89182208},"runtime":{"goroutines":33}},"filebeat":{"events":{"active":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"handles":{"open":4}}},"ecs.version":"1.6.0"}}
15

Are there any logs to be read?
You never showed what logs are being ingested?

What input or module are you using please show that configuration.

The last log line as well as previous log line shows that no input files are open and nothing has been read.

ctive":0},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"active":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system

{"harvester":{"open_files":0,"running":0}},
"libbeat":{"config":{"module":{"running":0}},
"output":{"events":{"active":0}},
"pipeline":{"clients":0,"events":{"active":0}}},

So to me it looks like file beat is working properly but there's no logs for it to read... Perhaps they've already been read and they will not be read again. You can fix that by cleaning out a data directory

Clean out this directory

C:\filebeat\data\registry\filebeat

Share the module and it input configuration.

16

@stephenb

This is module's list :

PS C:\filebeat> .\filebeat.exe modules list
Enabled:

Disabled:
activemq
apache
auditd
aws
awsfargate
azure
barracuda
bluecoat
cef
checkpoint
cisco
coredns
crowdstrike
cyberarkpas
cylance
elasticsearch
envoyproxy
f5
fortinet
gcp
google_workspace
haproxy
ibmmq
icinga
iis
imperva
infoblox
iptables
juniper
kafka
kibana
logstash
microsoft
misp
mongodb
mssql
mysql
mysqlenterprise
nats
netflow
netscout
nginx
o365
okta
oracle
osquery
panw
pensando
postgresql
proofpoint
rabbitmq
radware
redis
salesforce
santa
snort
snyk
sonicwall
sophos
squid
suricata
system
threatintel
tomcat
traefik
zeek
zookeeper
zoom
zscaler

Also in this path : C:\filebeat\data
there is just "meta.json" file and I deleted that file and another run my filebeat but it does not work correctly

17

So @baber1223

So you no modules enabled.

Please share you entire filebeat.yml.

We need to look at the input section.

To me looks like you have no modules and no inputs enabled or you have actual log files for filebeat to read

18

@stephenb

This is my yml file :

###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

# filestream is an input for collecting log messages from files.
- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - c:\ProgramData\Veeam\Backup\Svc.VeeamBackup
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  # Line filtering happens after the parsers pipeline. If you would like to filter lines
  # before parsers, use include_message parser.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #prospector.scanner.exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  #  level: debug
  #  review: 1

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "10.20.30.40:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["https://10.20.30.40:9200"]
  protocol: "https"

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  username: "elastic"
  password: "P@ssw0rd"
#  index: audit-db
  ssl:
    enabled: true
    ca_trusted_fingerprint: "23E9D473CE99BB2C6C331F43C94DA35B3096ABA83C844AB5CB04458F95AE83BC"
setup.template.name: "veeam-%{[agent.version]}"
setup.template.pattern: "veeam-%{[agent.version]}"
setup.dashboards.index: "veeam-*"
# ------------------------------ Logstash Output -------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    #  - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:


# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
19

Is that a file or folder?

20

@stephenb

This is file