I have a filebeat instance that sends event from CSV files to an ES cluster.
The ingest pipeline has the relevant mappings and everything works fine.
I'm trying to improve the pipeline for deduplication of event by adding a fingerprint to my processors:
processors: - fingerprint: fields: ["messageId"] target_field: "@metadata._id"
The ingest pipeline name is also declared in the yml config file.
My csv do not have any header for info.
Now when I launch the ingest I get the following in the logs:
ERROR [publisher] pipeline/client.go:106 Failed to publish event: failed to compute fingerprint: failed to find field [messageId] in event: key not found
I guess this is because filebeat doesn't know which csv column the messageId field is.
I don't seem to find how to declare my csv structure in the filebeat.yml file. Should I just add the
field config declaring my csv columns in the right order? Will it understand the order of columns? Not fully sure what to do here.