# Custom module for file beat - parsing a custom log fomat **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [December 9, 2017, 12:40am UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908 "2017-12-09T00:40:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![john.akash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john.akash/32/18707_2.png) [@john.akash](https://discuss.elastic.co/u/john.akash)\ **Post date:** [December 9, 2017, 12:40am UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908/1 "2017-12-09T00:40:06Z")
Hi Team, We do have a custom log in one of our infra and we are trying to push the data to ES using filebeat (don't want to use logstash). For better visualisation, we can create custom modules and make it happened. Could you please let us know how can we get this done? When we tried to run basic listing command f`ilebeat modules list` I am getting error like `Error in modules manager: modules management requires 'filebeat.config.modules.path' setting` Could you please provide some guidelines to get rid of these errors? Even we are trying to enable existing modules like apache we are getting the same error mentioned above. We are using filebeat 6 ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [December 11, 2017, 11:17am UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908/2 "2017-12-11T11:17:17Z")
Could you please share your configuration file? Please format it usin ``. It seems to me that your global options in`filebeat.configuration` is not properly set. The `modules.path` has to be set to the path to the available modules configurations. It is by default `modules.d/*.yml`. Filebeat modules can be created by checking out the repo ([https://github.com/elastic/beats](https://github.com/elastic/beats)). There is an extensive guide on how to create one: [https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html](https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html) Tweaking an existing modules does not require cloning the repo. It is sufficient to edit the Grok patterns and other processors in `pipeline.json` or `default.json` under the folder `ingest` of each module. ---
**Author:** ![john.akash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john.akash/32/18707_2.png) [@john.akash](https://discuss.elastic.co/u/john.akash)\ **Post date:** [December 11, 2017, 11:11pm UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908/3 "2017-12-11T23:11:16Z")
Hey @kvch, Thanks for the guide line, I was not using those line in the configuration file. Now I have added them and fix it. I am trying to parse a custom application logs to ES, that is the purpose of this exercise. Could you please let me know how can we create a custom module to parse the below given data? At least I need to have the dateTime from the messages ``` [DEBUG] 2017-12-07 03:57:27.064 [https-jsse-nio-8443-exec-10] RequestProcessor - Status >200 [INFO] 2017-12-07 04:00:41.015 [main] Application - Starting Application v0.0.1-SNAPSHOT on c62e0ddde6f5 with PID 141 (/etc/app1/bin/app1.jar started by app1 in /etc/app1/bin) [DEBUG] 2017-12-07 04:00:41.022 [main] Application - Running with Spring Boot v1.5.2.RELEASE, Spring v4.3.7.RELEASE [INFO] 2017-12-07 04:00:41.023 [main] Application - No active profile set, falling back to default profiles: default [INFO] 2017-12-07 04:00:44.265 [main] Application - Started Application in 3.824 seconds (JVM running for 4.755) [DEBUG] 2017-12-07 18:31:09.868 [https-jsse-nio-8443-exec-4] RestProcessor - https://127.0.0.1:8200/v1/auth/ldap/login/user1 [DEBUG] 2017-12-07 18:31:10.241 [https-jsse-nio-8443-exec-4] RequestProcessor - Status >200 ``` ---
**Author:** ![john.akash](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/john.akash/32/18707_2.png) [@john.akash](https://discuss.elastic.co/u/john.akash)\ **Post date:** [December 21, 2017, 10:38pm UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908/4 "2017-12-21T22:38:58Z")
Could you please provide an update on this? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [December 26, 2017, 9:22pm UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908/5 "2017-12-26T21:22:37Z")
For the above you need to put together the correct grok pattern. Best have a look at the docs here: [https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html](https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 23, 2018, 9:23pm UTC](https://discuss.elastic.co/t/custom-module-for-file-beat-parsing-a-custom-log-fomat/110908/6 "2018-01-23T21:23:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.