Filebeat index is getting created but with 0 documents

[anujaggarwal]

Hi

I am trying to index my custom log file using filebeat. I am successfully running filebeat with pre-built modules like mysql, nginx etc. But when I actually try to use it with my application specific log file, index is created with 0 documents.

I could not find anywhere in the filebeats document if there are any specific steps need to be taken to ensure indexing takes place for the custom log files.

I did not get any error when I setup filebeats or run filebeats post setup.

Below is the filebeat.yml:

filebeat.inputs:
- type: log
enabled: true
paths:
- /Applications/MAMP/htdocs/247around-adminp-aws/application/logs/log-2020-12-21.log
include_lines: ['^INFO', '^ERROR']
fields:
app_id: crm
filebeat.config.modules:
setup.template.settings:
index.number_of_shards: 1
path: ${path.config}/modules.d/*.yml
setup.kibana:
output.elasticsearch:
hosts: ["localhost:9200"]
processors:

As can be seen, it is majorly default .yml file with very minor changes.

My custom log file log-2020-12-21.php is:

INFO  - 2020-12-21 15:10:26 --> index Logging details have been captured for employee. Details are : Array
INFO  - 2020-12-21 15:10:36 --> editpartner partner_id:1
INFO  - 2020-12-21 15:10:36 --> SELECT DISTINCT service_id, brand, active
ERROR - 2020-12-21 15:10:36 --> Query error: Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'boloaaka.collateral.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
INFO  - 2020-12-21 15:10:36 -->  Database Error: A Database Error Occurred<br/>Array
ERROR - 2020-12-21 15:10:54 --> Query error: Expression #5 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'boloaaka.service_centres.district' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
INFO  - 2020-12-21 15:10:54 -->  Database Error: A Database Error Occurred<br/>Array
INFO  - 2020-12-21 23:53:21 --> Loginindex
INFO  - 2020-12-21 23:54:50 --> Loginindex
INFO  - 2020-12-21 23:55:42 --> Loginindex
INFO  - 2020-12-21 23:56:24 --> Loginindex

Index file is getting created with 0 documents:

Log file showing logs for filebeats setup and filebeats running:

Filebeat Log - 1 - Pastebin.com

Please help:

1. Why there are no error messages if something is wrong because of which documents are not getting indexed? I should be getting some error if things are not right.
2. How should I index my log file?
3. Where should I add pattern for my log file like key-value pair which would help me in searching the documents for relevant values later on?

Thanks for your help.

[shaunak (Shaunak Kashyap)]

When you configure Filebeat for your custom log file, can you temporarily set output.elasticsearch.enabled: false and output.console.enabled: true? Then start Filebeat and check if there are any events are being output to the console.

Shaunak

[anujaggarwal]

There was one error in my configuration file: log file name was incorrect. Log file extension was .PHP whereas in the log file, it was mentioned as .LOG which was causing the issue.

Once I fixed that, file was started getting indexed.