I want to collect logs from a remote service (Runnable.io) and send them onto our ELK stack using the Runnable CLI tool and Filebeat. The first part is easy but getting Filebeat to work has been trickier than I thought.
I've a tried a couple of different Filebeat configurations:
filebeat:
prospectors:
-
paths:
- "-"
document_type: apache
input_type: stdin
output:
logstash:
hosts: ["our-secret-hostname.logit.io:11144"]
… or:
filebeat:
prospectors:
- input_type: stdin
output:
console:
pretty: true
… and variations thereof.
But when I try it, Filebeat panics:
$ echo '127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /asdf HTTP/1.0" 200 2326' | filebeat -v -e -c filebeat.yml
2016/08/15 09:24:46.020376 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/08/15 09:24:46.026099 logstash.go:106: INFO Max Retries set to: 3
2016/08/15 09:24:46.683087 outputs.go:126: INFO Activated logstash as output plugin.
2016/08/15 09:24:46.683427 publish.go:288: INFO Publisher name: Tobiass-MacBook-Pro.local
2016/08/15 09:24:46.683583 async.go:78: INFO Flush Interval set to: 1s
2016/08/15 09:24:46.683603 async.go:84: INFO Max Bulk Size set to: 2048
2016/08/15 09:24:46.683700 beat.go:147: INFO Init Beat: filebeat; Version: 1.2.3
2016/08/15 09:24:46.684138 beat.go:173: INFO filebeat sucessfully setup. Start running.
2016/08/15 09:24:46.684419 registrar.go:68: INFO Registry file set to: /testdir/.filebeat
2016/08/15 09:24:46.684448 registrar.go:80: INFO Loading registrar data from /testdir/.filebeat
2016/08/15 09:24:46.685104 prospector.go:133: INFO Set ignore_older duration to 0
2016/08/15 09:24:46.685119 prospector.go:133: INFO Set close_older duration to 1h0m0s
2016/08/15 09:24:46.685124 prospector.go:133: INFO Set scan_frequency duration to 10s
2016/08/15 09:24:46.685130 prospector.go:93: INFO Input type set to: stdin
2016/08/15 09:24:46.685134 prospector.go:133: INFO Set backoff duration to 1s
2016/08/15 09:24:46.685139 prospector.go:133: INFO Set max_backoff duration to 10s
2016/08/15 09:24:46.685144 prospector.go:113: INFO force_close_file is disabled
2016/08/15 09:24:46.685184 prospector.go:143: INFO Starting prospector of type: stdin
2016/08/15 09:24:46.685321 crawler.go:78: INFO All prospectors initialised with 12 states to persist
2016/08/15 09:24:46.685331 spooler.go:77: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2016/08/15 09:24:46.685343 registrar.go:87: INFO Starting Registrar
2016/08/15 09:24:46.685364 publish.go:88: INFO Start sending events to output
2016/08/15 09:24:46.685640 log.go:113: INFO Harvester started for file: -
2016/08/15 09:24:46.686319 reader.go:87: INFO Reached end of file: /dev/stdin
2016/08/15 09:24:46.686335 log.go:157: INFO Read line error: EOF
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x10 pc=0x151a54]
goroutine 25 [running]:
panic(0x510b20, 0xc8200120d0)
/usr/local/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
github.com/elastic/beats/filebeat/harvester.(*Harvester).Harvest.func1(0xc8202c8800)
/private/tmp/filebeat-20160531-85714-13ut0se/beats-1.2.3/gopath/src/github.com/elastic/beats/filebeat/harvester/log.go:91 +0x54
github.com/elastic/beats/filebeat/harvester.(*Harvester).Harvest(0xc8202c8800)
/private/tmp/filebeat-20160531-85714-13ut0se/beats-1.2.3/gopath/src/github.com/elastic/beats/filebeat/harvester/log.go:158 +0xa7d
created by github.com/elastic/beats/filebeat/harvester.(*Harvester).Start
/private/tmp/filebeat-20160531-85714-13ut0se/beats-1.2.3/gopath/src/github.com/elastic/beats/filebeat/harvester/harvester.go:82 +0x35
What can I be doing wrong? I've tried this both on Mac and Ubuntu (Filebeat v1.2.3) but both show the same error.