Sending logs from filebeat(WIndows) to Logstash and Elasticsearch(RHEL)

Elastic Stack Beats
,
1

Hi, I have installed filebeat on my windows machine. I've enabled the systema nd logstash module.
Here is the filebeat.yml

- type: filestream

  # Unique ID among all inputs, an ID is required.
  id: my-filestream-id

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - C:\ProgramData\filebeat\logs\*
  fields:
    type: windows_log
  fields_under_root: true


filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

output.logstash:
  # The Logstash hosts
  hosts: ["logstash-ip:5044"]


 .\filebeat -e -c "C:\Program Files\Filebeat\filebeat.yml" test output
{"log.level":"info","@timestamp":"2023-04-21T12:47:01.489+0530","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [C:\\Program Files\\Filebeat] Config path: [C:\\Program Files\\Filebeat] Data path: [C:\\Program Files\\Filebeat\\data] Logs path: [C:\\Program Files\\Filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:47:01.490+0530","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: cea733d5-40bd-4c1e-adad-0041a5258c1f","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-21T12:47:04.522+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
logstash: LOGSTASH:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: LOGSTASH IP
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

Output for .\filebeat -e -c "C:\Program Files\Filebeat\filebeat.yml" -d "publish" command

.\filebeat -e -c "C:\Program Files\Filebeat\filebeat.yml" -d "publish"
{"log.level":"info","@timestamp":"2023-04-21T12:49:30.405+0530","log.origin":{"file.name":"instance/beat.go","file.line":724},"message":"Home path: [C:\\Program Files\\Filebeat] Config path: [C:\\Program Files\\Filebeat] Data path: [C:\\Program Files\\Filebeat\\data] Logs path: [C:\\Program Files\\Filebeat\\logs]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:30.407+0530","log.origin":{"file.name":"instance/beat.go","file.line":732},"message":"Beat ID: cea733d5-40bd-4c1e-adad-0041a5258c1f","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-21T12:49:33.470+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/provider_aws_ec2.go","file.line":81},"message":"read token request for getting IMDSv2 token returns empty: Put \"http://169.254.169.254/latest/api/token\": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.482+0530","log.logger":"add_cloud_metadata","log.origin":{"file.name":"add_cloud_metadata/add_cloud_metadata.go","file.line":102},"message":"add_cloud_metadata: hosting provider type not detected.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.484+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1096},"message":"Beat info","service.name":"filebeat","system_info":{"beat":{"path":{"config":"C:\\Program Files\\Filebeat","data":"C:\\Program Files\\Filebeat\\data","home":"C:\\Program Files\\Filebeat","logs":"C:\\Program Files\\Filebeat\\logs"},"type":"filebeat","uuid":"cea733d5-40bd-4c1e-adad-0041a5258c1f"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.485+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1105},"message":"Build info","service.name":"filebeat","system_info":{"build":{"commit":"a8dbc6c06381f4fe33a5dc23906d63c04c9e2444","libbeat":"8.7.0","time":"2023-03-23T00:44:06.000Z","version":"8.7.0"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.485+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1108},"message":"Go runtime info","service.name":"filebeat","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":8,"version":"go1.19.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.521+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1114},"message":"Host info","service.name":"filebeat","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-03-28T20:15:17+05:30","name":"DESKTOP-9MRJO59","ip":["fe80::42d:24da:df9a:3073","169.254.253.200","2405:201:d014:1b79:f392:e5e3:b66e:c61e","2405:201:d014:1b79:c1b1:5b6:9168:568e","fe80::7f1e:d347:54e8:7ba","192.168.29.224","fe80::64b7:4f34:ce0b:f79a","169.254.131.149","::1","127.0.0.1"],"kernel_version":"10.0.19041.2728 (WinBuild.160101.0800)","mac":["70:cd:0d:f7:d1:1e","70:cd:0d:f7:d1:1d","70:cd:0d:f7:d1:21"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"19044.2728"},"timezone":"IST","timezone_offset_sec":19800,"id":"fe2b1338-9913-4d24-b213-3d72415007ae"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.521+0530","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1143},"message":"Process info","service.name":"filebeat","system_info":{"process":{"cwd":"C:\\Program Files\\Filebeat","exe":"C:\\Program Files\\Filebeat\\filebeat.exe","name":"filebeat.exe","pid":28348,"ppid":18556,"start_time":"2023-04-21T12:49:30.226+0530"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-21T12:49:36.524+0530","log.origin":{"file.name":"instance/beat.go","file.line":297},"message":"Setup Beat: filebeat; Version: 8.7.0","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.924+0530","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: DESKTOP-9MRJO59","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.925+0530","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: ","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-21T12:49:38.928+0530","log.origin":{"file.name":"beater/filebeat.go","file.line":175},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.937+0530","log.origin":{"file.name":"instance/beat.go","file.line":486},"message":"filebeat start running.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.937+0530","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":145},"message":"Starting metrics logging every 30s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.944+0530","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\Program Files\\Filebeat\\data\\registry\\filebeat'. Active transaction id=58","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.947+0530","log.origin":{"file.name":"memlog/store.go","file.line":134},"message":"Finished loading transaction log file for 'C:\\Program Files\\Filebeat\\data\\registry\\filebeat'. Active transaction id=58","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-21T12:49:38.950+0530","log.origin":{"file.name":"beater/filebeat.go","file.line":307},"message":"Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.959+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":109},"message":"States Loaded from registrar: 2","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.959+0530","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":71},"message":"Loading Inputs: 1","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.960+0530","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":117},"message":"starting input, keys present on the config: [filebeat.inputs.0.enabled filebeat.inputs.0.fields.type filebeat.inputs.0.fields_under_root filebeat.inputs.0.id filebeat.inputs.0.paths.0 filebeat.inputs.0.type]","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.968+0530","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":148},"message":"Starting input (ID: 8353475235163233459)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.969+0530","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":113},"message":"Input 'filestream' starting","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.973+0530","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: logstash (log), logstash (slowlog)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-21T12:49:38.977+0530","log.logger":"cfgwarn","log.origin":{"file.name":"log/input.go","file.line":90},"message":"DEPRECATED: Log input. Use Filestream input instead.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.980+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":172},"message":"Configured paths: [c:\\programdata\\logstash\\logs\\logstash-plain*.log c:\\programdata\\logstash\\logs\\logstash-json*.log]","service.name":"filebeat","input_id":"982ce92f-3cf4-46cd-910d-edc26357b8f9","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.989+0530","log.logger":"input","log.origin":{"file.name":"log/input.go","file.line":172},"message":"Configured paths: [c:\\programdata\\logstash\\logs\\logstash-slowlog-plain*.log c:\\programdata\\logstash\\logs\\logstash-slowlog-json*.log]","service.name":"filebeat","input_id":"8c31a3c1-9ffe-46a9-80c7-6bc3602d7c05","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.996+0530","log.logger":"modules","log.origin":{"file.name":"fileset/modules.go","file.line":120},"message":"Enabled modules/filesets: system (syslog), system (auth)","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:38.998+0530","log.origin":{"file.name":"beater/crawler.go","file.line":155},"message":"Stopping Crawler","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.005+0530","log.origin":{"file.name":"beater/crawler.go","file.line":165},"message":"Stopping 1 inputs","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.006+0530","log.logger":"crawler","log.origin":{"file.name":"beater/crawler.go","file.line":170},"message":"Stopping input: 8353475235163233459","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::3997696-119261-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-4.ndjson","state-id":"native::3997696-119261-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::5177344-131288-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421.ndjson","state-id":"native::5177344-131288-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::4980736-28231-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-1.ndjson","state-id":"native::4980736-28231-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::2031616-60728-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-3.ndjson","state-id":"native::2031616-60728-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::7274496-119335-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-5.ndjson","state-id":"native::7274496-119335-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::2555904-46724-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-2.ndjson","state-id":"native::2555904-46724-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::1638400-68862-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-6.ndjson","state-id":"native::1638400-68862-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.007+0530","log.logger":"input.filestream","log.origin":{"file.name":"filestream/input.go","file.line":321},"message":"Reader was closed. Closing.","service.name":"filebeat","id":"my-filestream-id","source_file":"filestream::my-filestream-id::native::2097152-105535-10164368","path":"C:\\ProgramData\\filebeat\\logs\\filebeat-20230421-7.ndjson","state-id":"native::2097152-105535-10164368","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.039+0530","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":126},"message":"Input 'filestream' stopped (goroutine)","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.051+0530","log.logger":"input.filestream","log.origin":{"file.name":"compat/compat.go","file.line":134},"message":"Input 'filestream' stopped (runner)","service.name":"filebeat","id":"my-filestream-id","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.053+0530","log.origin":{"file.name":"beater/crawler.go","file.line":185},"message":"Crawler stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.055+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":132},"message":"Stopping Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.055+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":166},"message":"Ending Registrar","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.056+0530","log.logger":"registrar","log.origin":{"file.name":"registrar/registrar.go","file.line":137},"message":"Registrar stopped","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.059+0530","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":195},"message":"Total metrics","service.name":"filebeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":375,"time":{"ms":375}},"total":{"ticks":656,"time":{"ms":656},"value":656},"user":{"ticks":281,"time":{"ms":281}}},"info":{"ephemeral_id":"c733fccd-bb4d-4db0-981a-0e41172fca59","name":"filebeat","uptime":{"ms":8763},"version":"8.7.0"},"memstats":{"gc_next":23773864,"memory_alloc":12834648,"memory_sys":37293528,"memory_total":56562880,"rss":58028032},"runtime":{"goroutines":16}},"filebeat":{"events":{"active":398,"added":398,"done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}},"netflow":{"flows":0,"packets":{"dropped":0,"received":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops":0},"reloads":0,"scans":0},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"logstash","write":{"bytes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":398,"dropped":0,"failed":0,"filtered":0,"published":398,"retry":0,"total":398},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"cleanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":8},"handles":{"open":248}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.060+0530","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":196},"message":"Uptime: 8.7692897s","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.061+0530","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":163},"message":"Stopping metrics logging.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-21T12:49:39.062+0530","log.origin":{"file.name":"instance/beat.go","file.line":491},"message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-04-21T12:49:39.062+0530","log.origin":{"file.name":"instance/beat.go","file.line":1071},"message":"Exiting: Failed to start crawler: creating module reloader failed: error checking input configuration: No paths were defined for input accessing config","service.name":"filebeat","ecs.version":"1.6.0"}
Exiting: Failed to start crawler: creating module reloader failed: error checking input configuration: No paths were defined for input accessing config

WHere am i going wrong?

2

had to disable system module. and it started working. i mean only filebeat error has gone. on the logstash side when i run the command
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/first-pipeline.conf --config.reload.automatic
I am getting this

[INFO ] 2023-04-21 07:47:19.706 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/logstash/conf.d/first-pipeline.conf"}
[ERROR] 2023-04-21 07:47:19.707 [Agent thread] sourceloader - No configuration found in the configured sources.
3

Can you show the first-pipeline.conf content?

4

sure. Here it is

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => "http://elastic_ip:9200"
    index => "test"
    user => "user"
    password => "pwd"
  }
}
5

Here are the steps i am following, with the above config

  1. .\filebeat -e -c "C:\Program Files\Filebeat\filebeat.yml" test output
    Result shows
    logstash: logstaship:5044...
    connection...
    parse host... OK
    dns lookup... OK
    addresses:logstaship
    dial up... OK
    TLS... WARN secure connection disabled
    talk to server... OK
  2. Going to the windows services page and restarting filebeat service.
  3. Logstash is on RHEL so restarting that using systemctl.
    In logstash logs I'm getting address already in use.

am i missing something from filebeat side or logstash side?

6

This means you have already running LS process. Check with ps aux and kill the process.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.