Hello, I am new user of the elastic components. I am trying to get a FileBeats => Logstash => ElasticSearch pipeline working. I have been able to verify LS => ES working, but nothing is happening with FB => LS. I have checked the FileBeats log and can see that the harvester was started for the inputs. Logstash doesn't appear to do anything with the harvested data and I confirmed that no new indices show up in elasticsearch.
FileBeats.yml:
filebeat.inputs:
- type: log
enabled: true
paths:
- c:\elk\logs\*
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
output.logstash:
hosts: ["localhost:5044"]
index: "fb"
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
logstash.conf:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "logstash-beats"
#user => "elastic"
#password => "changeme"
}
}
I have verified that the ports are working with netstat and the configuration seems to be correct. There is output to the logs and seemingly without errors. I am using Postman and doing a "Get: http://localhost:9200/_cat/indices" to see if anything makes it into elasticsearch db. Can't figure out what I am doing wrong.
Please help! Thanks in advance!
Filebeats Log:
2019-08-20T11:56:12.136-0700 INFO instance/beat.go:606 Home path: [C:\elk\filebeat] Config path: [C:\elk\filebeat] Data path: [C:\elk\filebeat\data] Logs path: [C:\elk\filebeat\logs]
2019-08-20T11:56:12.137-0700 INFO instance/beat.go:614 Beat ID: ac845bac-bb74-4517-8732-80c6ef802e05
2019-08-20T11:56:12.141-0700 INFO [beat] instance/beat.go:902 Beat info {"system_info": {"beat": {"path": {"config": "C:\elk\filebeat", "data": "C:\elk\filebeat\data", "home": "C:\elk\filebeat", "logs": "C:\elk\filebeat\logs"}, "type": "filebeat", "uuid": "ac845bac-bb74-4517-8732-80c6ef802e05"}}}
2019-08-20T11:56:12.141-0700 INFO [beat] instance/beat.go:911 Build info {"system_info": {"build": {"commit": "6f0ec01a0e57fe7d4fd703b017fb5a2f6448d097", "libbeat": "7.3.0", "time": "2019-07-24T17:39:33.000Z", "version": "7.3.0"}}}
2019-08-20T11:56:12.141-0700 INFO [beat] instance/beat.go:914 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.12.4"}}}
2019-08-20T11:56:12.143-0700 INFO [beat] instance/beat.go:918 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-08-20T10:59:52.26-07:00","name":"vagrant-2012-r2","ip":["fe80::4c67:7188:66c1:e795/64","10.0.2.15/24","::1/128","127.0.0.1/8","fe80::5efe:a00:20f/128"],"kernel_version":"6.3.9600.19426 (winblue_ltsb_escrow.190715-1750)","mac":["08:00:27:e9:96:13","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19431"},"timezone":"PDT","timezone_offset_sec":-25200,"id":"42f11c3b-3062-4874-9ea3-50e0e08c4434"}}}
2019-08-20T11:56:12.146-0700 INFO [beat] instance/beat.go:947 Process info {"system_info": {"process": {"cwd": "C:\elk\filebeat", "exe": "C:\elk\filebeat\filebeat.exe", "name": "filebeat.exe", "pid": 3868, "ppid": 5620, "start_time": "2019-08-20T11:56:12.109-0700"}}}
2019-08-20T11:56:12.146-0700 INFO instance/beat.go:292 Setup Beat: filebeat; Version: 7.3.0
2019-08-20T11:56:12.147-0700 INFO [publisher] pipeline/module.go:97 Beat name: vagrant-2012-r2
2019-08-20T11:56:12.147-0700 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-08-20T11:56:12.147-0700 INFO instance/beat.go:421 filebeat start running.
2019-08-20T11:56:12.147-0700 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2019-08-20T11:56:12.148-0700 INFO registrar/registrar.go:145 Loading registrar data from C:\elk\filebeat\data\registry\filebeat\data.json
2019-08-20T11:56:12.148-0700 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2019-08-20T11:56:12.148-0700 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2019-08-20T11:56:12.148-0700 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-08-20T11:56:12.148-0700 INFO log/input.go:148 Configured paths: [c:\elk\logs*]
2019-08-20T11:56:12.148-0700 INFO input/input.go:114 Starting input of type: log; ID: 2112759327222265173
2019-08-20T11:56:12.148-0700 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2019-08-20T11:56:12.148-0700 INFO cfgfile/reload.go:171 Config reloader started
2019-08-20T11:56:12.149-0700 INFO cfgfile/reload.go:226 Loading of config files completed.
2019-08-20T11:56:12.156-0700 INFO log/harvester.go:253 Harvester started for file: c:\elk\logs\test.log
2019-08-20T11:56:15.053-0700 INFO add_cloud_metadata/add_cloud_metadata.go:347 add_cloud_metadata: hosting provider type not detected.
2019-08-20T11:56:42.874-0700 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":15,"time":{"ms":15}},"total":{"ticks":30,"time":{"ms":30},"value":30},"user":{"ticks":15,"time":{"ms":15}}},"handles":{"open":161},"info":{"ephemeral_id":"2dc9b8c8-f8f7-4020-9f67-a946240b4427","uptime":{"ms":30028}},"memstats":{"gc_next":4523760,"memory_alloc":2700912,"memory_total":7650424,"rss":23592960},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}