Using Modules and Custom logs

[bmclellan (Barry)]

Hello,

Is it possible to use the modules that come with filebeat and create your own custom logs as well?
I attempted to follow the file structure of the 'iis' module and create my own, but it doesn't seem to be processing the pipeline I have made (despite it working in the pipeline/_simulation)

Thanks!

[bmclellan (Barry)]

Sigh, after a week, I stumbled onto a thread that said to do this from windows:
./filebeat setup --pipelines -modules="VPApp"

Apparently you need to do this if you change your grok.
Is this documented for filebeats windows development? I can't find it anywhere!

[kvch (Noémi Ványi)]

Unfortunately, the documentation on this topic is hard to find. You can read about it here: https://www.elastic.co/guide/en/logstash/current/use-ingest-pipelines.html#use-ingest-pipelines

Also, you can find a guide about developing your own modules: https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html

Let me know if you need more help.

[bmclellan (Barry)]

Hello,

I've somewhat got the custom modules figured out on Windows, however I have 2 questions still.
1 - I created a custom module and after running the pipeline update in PS, I can see it in my pipeline on Kibana (GET /_ingest/pipeline?pretty). When I go to 'discover' in Kibana I can see the individual fields, however when I check the filebeat index, I can't see the fields created there. I have one module using the processor "grok" which looks to have updated the indext, however using processor "dissect" for my new module, it is not updating the filebeat index. Am I missing a step?

2 - I looked at the instructions creating a filebeat module in Linux and it looked pretty good, however I want to do this in Windows instead (which did not work out for me). I ended up just coping the 'iis' module flow and creating my own module based on that. My question; how do I add my fields into the index template? In Linux is creates your own fields.yml per module however in windows it looks like I should be exporting the already uploaded template:
PS C:\Program Files\Filebeat>
.\filebeat.exe export template --es.version 7.6.1 | Out-File -Encoding UTF8 filebeat.template.json

Updating that and then importing it back in
PS > Invoke-RestMethod -Method Put -ContentType "application/json" -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat-7.6.1

Is that correct?

Once done should I clear out my indexes before restarting the filebeat module?
Clear out all indexes:
Invoke-RestMethod -Method Delete http://localhost:9200/filebeat-*

Thanks
Barry

[bmclellan (Barry)]

@bmclellan

the custom modules figured out on Windows, however I have 2 questions still.
1 - I created a custom module and after running the pipeline update in PS, I can see it in my pipeline on Kibana (GET /_ingest/pipeline?pretty). When I go to 'discover' in Kibana I can see the individual fields, however when I check the filebeat index, I can't see the fields created there. I have one module using the processor "grok" which looks to have updat

answering my own question #1, dissect is not creating index fields, grok is. Switch to grok, all is ok now.