Can FileBeat parse csv file and send the data in fields to elastic search directly?

Elastic Stack Beats
1

I am listening to a log file (which is a CSV file) using FileBeat and sending the data to elasticsearch instance. I am using the default configurations for FileBeat as well as elasticsearch. It works but my data is shown in elastic in a field called message.
e.g.

"input_type": "log",
"message": "1481250458524,263,dummy request 3,200,OK,Thread Group 2 2-2,text,true,,114,0,10,15,39,0,2",
"offset": 143092,

How can I make it process message into different fields before sending it to elastic?

2

Filebeat is designed to be light-weight, and therefore supports limited parsing. I do not believe parsing of CSV event is currently supported in Filebeat, but you can do this by sending the data to either Logstash or an Elasticsearch ingest node.

3

Thanks, I'll try the elasticsearch ingest node. Do you have any guide on that?

4

The documentation is available here: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html

You can also look at the following blog posts:

5

Thanks Christian. I'll try that out.

Best regards,
Dina.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.