Using Modules and Custom logs


Is it possible to use the modules that come with filebeat and create your own custom logs as well?
I attempted to follow the file structure of the 'iis' module and create my own, but it doesn't seem to be processing the pipeline I have made (despite it working in the pipeline/_simulation)


Sigh, after a week, I stumbled onto a thread that said to do this from windows:
./filebeat setup --pipelines -modules="VPApp"

Apparently you need to do this if you change your grok.
Is this documented for filebeats windows development? I can't find it anywhere!

Unfortunately, the documentation on this topic is hard to find. You can read about it here:

Also, you can find a guide about developing your own modules:

Let me know if you need more help.


I've somewhat got the custom modules figured out on Windows, however I have 2 questions still.
1 - I created a custom module and after running the pipeline update in PS, I can see it in my pipeline on Kibana (GET /_ingest/pipeline?pretty). When I go to 'discover' in Kibana I can see the individual fields, however when I check the filebeat index, I can't see the fields created there. I have one module using the processor "grok" which looks to have updated the indext, however using processor "dissect" for my new module, it is not updating the filebeat index. Am I missing a step?

2 - I looked at the instructions creating a filebeat module in Linux and it looked pretty good, however I want to do this in Windows instead (which did not work out for me). I ended up just coping the 'iis' module flow and creating my own module based on that. My question; how do I add my fields into the index template? In Linux is creates your own fields.yml per module however in windows it looks like I should be exporting the already uploaded template:
PS C:\Program Files\Filebeat>
.\filebeat.exe export template --es.version 7.6.1 | Out-File -Encoding UTF8 filebeat.template.json

Updating that and then importing it back in
PS > Invoke-RestMethod -Method Put -ContentType "application/json" -InFile filebeat.template.json -Uri http://localhost:9200/_template/filebeat-7.6.1

Is that correct?

Once done should I clear out my indexes before restarting the filebeat module?
Clear out all indexes:
Invoke-RestMethod -Method Delete http://localhost:9200/filebeat-*


answering my own question #1, dissect is not creating index fields, grok is. Switch to grok, all is ok now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.