Expected log for successful FileBeats => Logstash integration

Hello!

Can anyone confirm that this is the expected output of filebeats when it successfully harvests and ships the associated data to logstash?

filebeats.yml:

filebeat.inputs:
- type: log
  paths:
    - c:\elk\logs\test.log

output.logstash:
  hosts: ["localhost:5044"]

logstash conf:

input {
    beats {
        port => "5044"
    }
}

output { 
 stdout { codec => rubydebug } 
}

filebeats log output:

2019-08-22T06:52:15.307-0700	INFO	instance/beat.go:606	Home path: [C:\elk\filebeat] Config path: [C:\elk\filebeat] Data path: [C:\elk\filebeat\data] Logs path: [C:\elk\filebeat\logs]
2019-08-22T06:52:15.309-0700	INFO	instance/beat.go:614	Beat ID: ac845bac-bb74-4517-8732-80c6ef802e05
2019-08-22T06:52:15.309-0700	INFO	[beat]	instance/beat.go:902	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\elk\\filebeat", "data": "C:\\elk\\filebeat\\data", "home": "C:\\elk\\filebeat", "logs": "C:\\elk\\filebeat\\logs"}, "type": "filebeat", "uuid": "ac845bac-bb74-4517-8732-80c6ef802e05"}}}
2019-08-22T06:52:15.309-0700	INFO	[beat]	instance/beat.go:911	Build info	{"system_info": {"build": {"commit": "6f0ec01a0e57fe7d4fd703b017fb5a2f6448d097", "libbeat": "7.3.0", "time": "2019-07-24T17:39:33.000Z", "version": "7.3.0"}}}
2019-08-22T06:52:15.309-0700	INFO	[beat]	instance/beat.go:914	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2019-08-22T06:52:15.314-0700	INFO	[beat]	instance/beat.go:918	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-08-21T20:52:23.4-07:00","name":"vagrant-2012-r2","ip":["fe80::4c67:7188:66c1:e795/64","10.0.2.15/24","::1/128","127.0.0.1/8","2001:0:5cf2:8c02:1c54:153c:f5ff:fdf0/64","fe80::1c54:153c:f5ff:fdf0/64","fe80::5efe:a00:20f/128"],"kernel_version":"6.3.9600.19426 (winblue_ltsb_escrow.190715-1750)","mac":["08:00:27:e9:96:13","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19431"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"42f11c3b-3062-4874-9ea3-50e0e08c4434"}}}
2019-08-22T06:52:15.315-0700	INFO	[beat]	instance/beat.go:947	Process info	{"system_info": {"process": {"cwd": "C:\\elk\\filebeat", "exe": "C:\\elk\\filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 3560, "ppid": 2960, "start_time": "2019-08-22T06:52:15.268-0700"}}}
2019-08-22T06:52:15.315-0700	INFO	instance/beat.go:292	Setup Beat: filebeat; Version: 7.3.0
2019-08-22T06:52:15.315-0700	INFO	[publisher]	pipeline/module.go:97	Beat name: vagrant-2012-r2
2019-08-22T06:52:15.316-0700	WARN	beater/filebeat.go:152	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-08-22T06:52:15.316-0700	INFO	instance/beat.go:421	filebeat start running.
2019-08-22T06:52:15.316-0700	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2019-08-22T06:52:15.317-0700	INFO	registrar/registrar.go:145	Loading registrar data from C:\elk\filebeat\data\registry\filebeat\data.json
2019-08-22T06:52:15.317-0700	INFO	registrar/registrar.go:152	States Loaded from registrar: 1
2019-08-22T06:52:15.317-0700	WARN	beater/filebeat.go:368	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-08-22T06:52:15.317-0700	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-08-22T06:52:15.317-0700	INFO	log/input.go:148	Configured paths: [c:\elk\logs\test.log]
2019-08-22T06:52:15.317-0700	INFO	input/input.go:114	Starting input of type: log; ID: 2903726485811444168 
2019-08-22T06:52:15.317-0700	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2019-08-22T06:52:15.317-0700	INFO	log/harvester.go:253	Harvester started for file: c:\elk\logs\test.log
2019-08-22T06:52:46.155-0700	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":15,"time":{"ms":15}},"total":{"ticks":30,"time":{"ms":30},"value":0},"user":{"ticks":15,"time":{"ms":15}}},"handles":{"open":141},"info":{"ephemeral_id":"b4ca2c98-9110-441c-adfa-d1c03b8bc488","uptime":{"ms":30045}},"memstats":{"gc_next":7761456,"memory_alloc":4465120,"memory_total":7445408,"rss":22880256},"runtime":{"goroutines":26}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":4}}}}}

logstash result:
ls
(not receiving anything - seemingly)

The configuration for both filebeat and logstash look fine but i see a problem

Beats input: Starting input listener <:address=>"localhost:5043">

From filebeat, you are sending to port 5044 but the listener on logstash is for port 5043

You can pass the config file using command line. I am using logstash on Ubuntu

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

[INFO ] 2019-08-23 16:27:00.395 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>6, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>750, :thread=>"#<Thread:0x28701b13 run>"}
[INFO ] 2019-08-23 16:27:01.161 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2019-08-23 16:27:01.173 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2019-08-23 16:27:01.383 [[main]<beats] Server - Starting server on port: 5044

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.