# Question about creating Brobeat
**URL:** https://discuss.elastic.co/t/question-about-creating-brobeat/70025
**Category:** Beats
**Created:** [December 26, 2016, 11:52pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025 "2016-12-26T23:52:31Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [December 26, 2016, 11:52pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/1 "2016-12-26T23:52:31Z")
I am working on a new beat called brobeat here - [https://github.com/blacktop/brobeat](https://github.com/blacktop/brobeat).
However, it looks like what I want is to build a filebeat module? Does it make sense for me to create a copy/fork of filebeat and add my bro-module?
All of the community beats seem to consume some API. What I need is to read log files off the disk and ingest them into elasticsearch and 'grok pattern match them' and maybe rename a few fields etc etc.
I am trying to avoid using Logstash if I can for now.
So I guess that libbeat doesn't do a logstash like function, it just ships logs, but can tell an Elastic Ingest node what pipeline to use?
So I would have to define a bunch of ingest-pipelines?
What do you think my path should be?
I would also love to talk to someone about how the filebeat modules integrate into filebeat. I don't see mentions of them in the filebeat golang code? Maybe it is happening deeper in libbeat somewhere?
Any help MUCH appreciated.
Thanks!
---
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [December 27, 2016, 12:53am UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/2 "2016-12-27T00:53:47Z")
It looks like there is new functionality for filebeat that IS called filebeat-modules - [https://github.com/elastic/beats/pull/3158](https://github.com/elastic/beats/pull/3158)
It sounds like that is what I am trying to build? Are you going to make a cookiecutter for filebeat modules and have a way to install them like plugins to filebeat?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 27, 2016, 4:35am UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/3 "2016-12-27T04:35:45Z")
Why not just use filebeat and an ingest node?
Rebuilding filebeat just for one log type seems like overkill 🙂
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [December 27, 2016, 8:34am UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/4 "2016-12-27T08:34:41Z")
The Filebeat modules will contain Ingest Node configurations + prospector configuraiton + fields defintions + docs + etc. Here is an example for nginx access logs: [https://github.com/elastic/beats/tree/master/filebeat/module/nginx/access](https://github.com/elastic/beats/tree/master/filebeat/module/nginx/access)
It sounds like what you want to build will match FB modules quite well, but they are currently heavy work in progress (we're just passing the prototyping phase), so it's a bit early to contribute to them.
But you can already create the Ingest Node pipeline configuration and load it manually or with a script into Elasticsearch. What Filebeat modules will bring is a little bit of automation around loading all the necessary files.
---
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [December 27, 2016, 11:01pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/5 "2016-12-27T23:01:56Z")
Do you have any timeline or a feature-branch I could watch in the mean time?
Very exciting stuff!
---
### Author:  [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [December 28, 2016, 11:52pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/6 "2016-12-28T23:52:28Z")
Prototype is in master branch. Currently it's a python based filebeat wrapper [filebeat.py](https://github.com/elastic/beats/blob/master/filebeat/filebeat.py). Don't expect this to be the final outcome. The prototype allows us to mostly play/change the feature without much effort, to develop a good idea how the final module support will work.
---
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [December 29, 2016, 2:05am UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/7 "2016-12-29T02:05:20Z")
I have a question about filebeat modules and ingest nodes. BroIDS logs have a bunch of boiler plate headers that include log field names/types.
Is there a way to parse that? I tried to generate patterns here - [https://github.com/blacktop/brobeat/blob/master/logstash/patterns/generated-bro](https://github.com/blacktop/brobeat/blob/master/logstash/patterns/generated-bro), but what I don't understand is what happens when an optional field is not there? I assume the grok pattern fails?
Is there a way either in the ingest.json grok to label a field as optional or have it parse the log file header to see what fields are actually there?
This has me leaning back to the idea of a full beat that parses and understands the bro-log format?
Thanks for all the quick and insightful responses!
I have also taken a first stab at creating the bro filebeat modules here - [https://github.com/blacktop/brobeat/tree/master/module/bro](https://github.com/blacktop/brobeat/tree/master/module/bro)
I haven't tested them yet.
---
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [December 29, 2016, 6:11pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/8 "2016-12-29T18:11:02Z")
I tested out the logstash conf/patterns last night and they were failing to parse the http.logs 😢
Also I tried just using the logstash default bro grok-patterns and those also failed. I think it is something wrong with the way bro generates it's logs that the fields are not static and change a lot depending on what plugin/modules you have enabled.
So this has me leaning again more towards a full beat or maybe I just am not very good at grok?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [January 3, 2017, 8:36am UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/9 "2017-01-03T08:36:22Z")
Perhaps it is worth to post a question in the Logstash forum about the bro patterns not working. I would hope that Logstash can actually deal with almost all cases somehow 🙂
For the separate beat: It sounds kind of overkill to create a beat for in case it is still a log file in the common sense that logs are written line by line. But I don't know the details about the Bro NSM logs.
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [January 4, 2017, 8:02am UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/10 "2017-01-04T08:02:42Z")
> [@blacktop](#):
>
> Is there a way either in the ingest.json grok to label a field as optional or have it parse the log file header to see what fields are actually there?
There's several tricks you can use:
- enclose fields in `(...)?`to make them optional
- use multiple patterns, which are tried in order
- use on\_failure modes
- use conditionals in Logstash
For the ingest node, I recommend playing with the simulate API to figure it out: [Simulate Pipeline API | Elasticsearch Guide [5.1] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.1/simulate-pipeline-api.html)
---
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [January 4, 2017, 6:44pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/11 "2017-01-04T18:44:26Z")
So I will look into it again, but I still don't understand how it can work.
Let me give you an example:
fields: (ts:double, required), (ip:ip, required) (port:int, optional) (f1:int, optional)(f2:int, optional)(f3:int, optional)(f4:int, optional)
what if in one log there is f1 and f4 and then in another there is f2, f4.
How would logstash or a ingest node's grok pattern know to label the two int fields as (f1,f4) or (f2,f4) ?
---
### Author:  [@blacktop](https://discuss.elastic.co/u/blacktop)
#### Post date: [January 4, 2017, 7:17pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/12 "2017-01-04T19:17:02Z")
I created a question in the Logstash topic here: [Logstash-patterns-core BRO\_\* not working?](https://discuss.elastic.co/t/logstash-patterns-core-bro--not-working/70595?u=blacktop)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 16, 2017, 11:52pm UTC](https://discuss.elastic.co/t/question-about-creating-brobeat/70025/13 "2017-01-16T23:52:34Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.