Handle Error During Filebeat Ingestion

Elastic Stack Beats
1

Hi Team,

We had a requirement that when Filebeat reads the records from CSV files from the directory below, it should capture any records that encounter errors . Is there a method available to redirect all error-related records to separate .err files instead of capturing them in the Filebeat log files?

 paths:
   - /cbdata/elastic/cb4lv1/cb4transv4/*.csv # path to your CSV file

Note:- If we made log level in debug mode it will capture error in same log file but log file size will be huge.

Thanks,
Debasis

2

Any advise on the above request.

Thanks,
Debasis

3

It is not really clear what exactly you want to do and what is your issue.

Can you provide more context?

You want to exclude error lines from your csv? Or capture just the error lines?

4

Yes, during processing of csv files the error records should capture in .err files with failure reason so that support team later will go-through those records.

Thanks,
Debasis

5

I'm sorry, but what are those .err files, what is generating it? Are you reading them with filebeat or you are talking about errors from the filebeat process? It is still not clear.

In the configuration you shared you are only looking to .csv files, so it is not clear what those .err files are.

Please provide context and examples.

6

Hi @leandrojmp , The requirement is as below.

We will receive customer data in CSV files, which will be processed through Filebeat. In cases where records in the CSV files cannot be processed by Filebeat, instead of printing those records in the Filebeat log, we aim to redirect them to a separate file called filebeat.err. This file will exclusively contain the error records.

Thanks,
Debasis

7

Now I got it.

Unfortunately this is not possible.

8

Thanks for confirmation. Is there any way to find those error records from csv files after processing through filebeat.

Thanks,
Debasis