# Filebeat fingerprint doesn't understand columns from csv files

**URL:** <https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 25, 2021, 2:06am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427 "2021-08-25T02:06:51Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Baygon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baygon/32/91792_2.png) [@Baygon](https://discuss.elastic.co/u/Baygon)\
**Post date:** [August 25, 2021, 2:06am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427/1 "2021-08-25T02:06:51Z")

</div>

Hi,

I have a filebeat instance that sends event from CSV files to an ES cluster.  
The ingest pipeline has the relevant mappings and everything works fine.

I'm trying to improve the pipeline for deduplication of event by adding a fingerprint to my processors:

```auto
processors:
  - fingerprint:
      fields: ["messageId"]
      target_field: "@metadata._id"

```

The ingest pipeline name is also declared in the yml config file.  
My csv do not have any header for info.

Now when I launch the ingest I get the following in the logs:

> ERROR [publisher] pipeline/client.go:106 Failed to publish event: failed to compute fingerprint: failed to find field [messageId] in event: key not found

I guess this is because filebeat doesn't know which csv column the messageId field is.  
I don't seem to find how to declare my csv structure in the filebeat.yml file. Should I just add the `field` config declaring my csv columns in the right order? Will it understand the order of columns? Not fully sure what to do here.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 25, 2021, 2:50am UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427/2 "2021-08-25T02:50:06Z")

</div>

I don't think the `decode_csv_fields` processor works the way you think.

> It will output the values as an array of strings

Perhaps dissect processor would be a better fit.

> **[Dissect strings | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/dissect.html)**

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [August 25, 2021, 7:20pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427/3 "2021-08-25T19:20:55Z")

</div>

> [@Baygon](#):
>
> I guess this is because filebeat doesn't know which csv column the messageId field is.

If you are doing the csv parsing in Ingest Node then those fields will not exist when Filebeat is processing the event. You could use the Elasticsearch [fingerprint](https://www.elastic.co/guide/en/elasticsearch/reference/7.15/fingerprint-processor.html) processor instead.

Or move the csv parsing over to Filebeat with `decode_csv_fields` and `extract_array`. Then apply `fingerprint` afterward.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [August 25, 2021, 7:44pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427/4 "2021-08-25T19:44:43Z")

</div>

Ahh thanks @andrewkroh I was wondering how to use `decode_csv_fields` because it just returns an array, but then get to the actual field so you use the `extract_array` Good to know!

---

<div class="post-metadata">

**Author:** ![Baygon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baygon/32/91792_2.png) [@Baygon](https://discuss.elastic.co/u/Baygon)\
**Post date:** [August 27, 2021, 2:44pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427/5 "2021-08-27T14:44:35Z")

</div>

indeed using fingerprint upstream on the ES node itself is definitely going to work and do the trick, thanks for the tip!  
I think it would cost less than using a processor in filebeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 24, 2021, 4:45pm UTC](https://discuss.elastic.co/t/filebeat-fingerprint-doesnt-understand-columns-from-csv-files/282427/6 "2021-09-24T16:45:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.