Custom Input within Filebeat?

[thebenwaters (Ben Waters)]

I'm trying to create a beat for Okta.

https://github.com/forter/oktabeat

The only issue is it has to be pull and there's no way to track state.

I wanted to try and hijack off of Filebeat's registar to be able to track state. But I seem to be running into issues. I created a new input, but I can't seem to get filebeat to detect the input. It seems maybe the init function isn't called? I'm thinking about just trying to implement registrar outside of filebeat

2019-05-16T19:31:53.150+0300 INFO instance/beat.go:294 Setup Beat: oktabeat; Version: 8.0.0
2019-05-16T19:31:53.151+0300 INFO [publisher] pipeline/module.go:97 Beat name: shoco-forter.local
2019-05-16T19:31:53.151+0300 ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /Users/benwaters/go/src/github.com/forter/oktabeat/module
2019-05-16T19:31:53.151+0300 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2019-05-16T19:31:53.151+0300 INFO instance/beat.go:416 oktabeat start running.
2019-05-16T19:31:53.152+0300 INFO registrar/registrar.go:145 Loading registrar data from /Users/benwaters/go/src/github.com/forter/oktabeat/data/registry/filebeat/data.json
2019-05-16T19:31:53.152+0300 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:72 Loading Inputs: 1
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:76 inputConfig &{{{%!s(*ucfg.Config=&{{{0xc00024cdb0} inputs} 0xc000254000 0xc00030e440})} 0} %!s(*ucfg.Meta=&{_meta/example.yml}) %!s(*ucfg.fields=&{map[api_key:0xc000162370 okta_domain:0xc00019a7c0 type:0xc00019a780] })}
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:117 Enabled %!s(bool=true)
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:121 Connecting pipeline
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:141 Stopping Crawler
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:151 Stopping 0 inputs
2019-05-16T19:31:53.152+0300 INFO crawler/crawler.go:167 Crawler stopped
2019-05-16T19:31:53.152+0300 INFO registrar/registrar.go:367 Stopping Registrar
2019-05-16T19:31:53.152+0300 INFO registrar/registrar.go:293 Ending Registrar
2019-05-16T19:31:53.180+0300 INFO [monitoring] log/log.go:153 Uptime: 38.984906ms
2019-05-16T19:31:53.180+0300 INFO [monitoring] log/log.go:130 Stopping metrics logging.
2019-05-16T19:31:53.180+0300 INFO instance/beat.go:426 oktabeat stopped.
2019-05-16T19:31:53.180+0300 ERROR instance/beat.go:875 Exiting: Error while initializing input: Error creating input. No such input type exist: 'okta'
Exiting: Error while initializing input: Error creating input. No such input type exist: 'okta'

[adrisr (Adrian Serrano)]

@thebenwaters

I created a new input, but I can't seem to get filebeat to detect the input. It seems maybe the init function isn't called?

In golang, a package needs to be imported, otherwise it won't be built into the binary and its init method won't be called.

You need to run mage update in the filebeat directory so that an import for your new input gets added to include/list.go.

@thebenwaters

I'm thinking about just trying to implement registrar outside of filebeat

I'm not familiar with Okta, but it feels like maybe you're trying to fit functionality into Filebeat that isn't well suited for it. Have you considered creating your own Beat using libbeat?

Have a look at our Creating a New Beat guide if you haven't already.