Filestream processing of CSV files

Elastic Stack Elasticsearch
1

Hi all,

I'm slowly migrating over to filestream to process some log files and have data coming in and stored into the message field. Here is an example:

ec26.515d.ebcf,someUser,GSS-A-1FL-122,SD35

What I would like to do now is process that data and put it into different fields separated by the comma. Ideally, it would go into the following fields:

macAddress, User, AP, SSID

I think this would be handled by a processor but haven't had any luck with I've put together so far with this:

  • decode_csv_fields:
    fields:
    message: connectionData

This moves the data into a field called connectionData as an array of the values. If I want to break the values in the message field into new fields separated by the comma is there a different format to the processor I should be using?

Thanks for your help!

2

Hi @rsaeks

Writing to separate fields is not how the decode_csv_fields processor works

The decode_csv_fields processor decodes fields containing records in comma-separated format (CSV). It will output the values as an array of strings. This processor is available for Filebeat.

Perhaps dissect will work for what you want, it is also very efficient.

1 Like
3

@stephenb - Thank you for the info about using dissect as a processor! That is a much better approach and a whole lot quicker (plus easy to setup)

I appreciate sharing and will be using that one :slight_smile: