Custom module for file beat - parsing a custom log fomat

Elastic Stack Beats
1

Hi Team,

We do have a custom log in one of our infra and we are trying to push the data to ES using filebeat (don't want to use logstash).

For better visualisation, we can create custom modules and make it happened. Could you please let us know how can we get this done?

When we tried to run basic listing command

filebeat modules list

I am getting error like

Error in modules manager: modules management requires 'filebeat.config.modules.path' setting

Could you please provide some guidelines to get rid of these errors?

Even we are trying to enable existing modules like apache we are getting the same error mentioned above.

We are using filebeat 6

2

Could you please share your configuration file? Please format it usin </>.
It seems to me that your global options infilebeat.configuration is not properly set. The modules.path has to be set to the path to the available modules configurations. It is by default modules.d/*.yml.

Filebeat modules can be created by checking out the repo (https://github.com/elastic/beats). There is an extensive guide on how to create one: https://www.elastic.co/guide/en/beats/devguide/current/filebeat-modules-devguide.html

Tweaking an existing modules does not require cloning the repo. It is sufficient to edit the Grok patterns and other processors in pipeline.json or default.json under the folder ingest of each module.

1 Like
3

Hey @kvch,

Thanks for the guide line, I was not using those line in the configuration file. Now I have added them and fix it.

I am trying to parse a custom application logs to ES, that is the purpose of this exercise. Could you please let me know how can we create a custom module to parse the below given data?

At least I need to have the dateTime from the messages

[DEBUG] 2017-12-07 03:57:27.064 [https-jsse-nio-8443-exec-10] RequestProcessor - Status >200
[INFO ] 2017-12-07 04:00:41.015 [main] Application - Starting Application v0.0.1-SNAPSHOT on c62e0ddde6f5 with PID 141 (/etc/app1/bin/app1.jar started by app1 in /etc/app1/bin)
[DEBUG] 2017-12-07 04:00:41.022 [main] Application - Running with Spring Boot v1.5.2.RELEASE, Spring v4.3.7.RELEASE
[INFO ] 2017-12-07 04:00:41.023 [main] Application - No active profile set, falling back to default profiles: default
[INFO ] 2017-12-07 04:00:44.265 [main] Application - Started Application in 3.824 seconds (JVM running for 4.755)
[DEBUG] 2017-12-07 18:31:09.868 [https-jsse-nio-8443-exec-4] RestProcessor - https://127.0.0.1:8200/v1/auth/ldap/login/user1
[DEBUG] 2017-12-07 18:31:10.241 [https-jsse-nio-8443-exec-4] RequestProcessor - Status >200
4

Could you please provide an update on this?

5

For the above you need to put together the correct grok pattern. Best have a look at the docs here: https://www.elastic.co/guide/en/elasticsearch/reference/master/grok-processor.html

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.