Filebeat index is getting created but with 0 documents

Hi

I am trying to index my custom log file using filebeat. I am successfully running filebeat with pre-built modules like mysql, nginx etc. But when I actually try to use it with my application specific log file, index is created with 0 documents.

I could not find anywhere in the filebeats document if there are any specific steps need to be taken to ensure indexing takes place for the custom log files.

I did not get any error when I setup filebeats or run filebeats post setup.

Below is the filebeat.yml:

filebeat.inputs:
- type: log
enabled: true
paths:
- /Applications/MAMP/htdocs/247around-adminp-aws/application/logs/log-2020-12-21.log
include_lines: ['^INFO', '^ERROR']
fields:
app_id: crm
filebeat.config.modules:
setup.template.settings:
index.number_of_shards: 1
path: ${path.config}/modules.d/*.yml
setup.kibana:
output.elasticsearch:
hosts: ["localhost:9200"]
processors:

As can be seen, it is majorly default .yml file with very minor changes.

My custom log file log-2020-12-21.php is:

INFO  - 2020-12-21 15:10:26 --> index Logging details have been captured for employee. Details are : Array
INFO  - 2020-12-21 15:10:36 --> editpartner partner_id:1
INFO  - 2020-12-21 15:10:36 --> SELECT DISTINCT service_id, brand, active
ERROR - 2020-12-21 15:10:36 --> Query error: Expression #1 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'boloaaka.collateral.id' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
INFO  - 2020-12-21 15:10:36 -->  Database Error: A Database Error Occurred<br/>Array
ERROR - 2020-12-21 15:10:54 --> Query error: Expression #5 of SELECT list is not in GROUP BY clause and contains nonaggregated column 'boloaaka.service_centres.district' which is not functionally dependent on columns in GROUP BY clause; this is incompatible with sql_mode=only_full_group_by
INFO  - 2020-12-21 15:10:54 -->  Database Error: A Database Error Occurred<br/>Array
INFO  - 2020-12-21 23:53:21 --> Loginindex
INFO  - 2020-12-21 23:54:50 --> Loginindex
INFO  - 2020-12-21 23:55:42 --> Loginindex
INFO  - 2020-12-21 23:56:24 --> Loginindex

Index file is getting created with 0 documents:

Log file showing logs for filebeats setup and filebeats running:

Please help:

  1. Why there are no error messages if something is wrong because of which documents are not getting indexed? I should be getting some error if things are not right.
  2. How should I index my log file?
  3. Where should I add pattern for my log file like key-value pair which would help me in searching the documents for relevant values later on?

Thanks for your help.

When you configure Filebeat for your custom log file, can you temporarily set output.elasticsearch.enabled: false and output.console.enabled: true? Then start Filebeat and check if there are any events are being output to the console.

Shaunak

There was one error in my configuration file: log file name was incorrect. Log file extension was .PHP whereas in the log file, it was mentioned as .LOG which was causing the issue.

Once I fixed that, file was started getting indexed.