# CSV filter plugin and changing column headers

**URL:** https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493
**Category:** Logstash
**Created:** [January 17, 2020, 4:24pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493 "2020-01-17T16:24:50Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![mhyager](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@mhyager](https://discuss.elastic.co/u/mhyager)
#### Post date: [January 17, 2020, 4:24pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493/1 "2020-01-17T16:24:50Z")

</div>

I have a set of CSV files which have similar but not identical headers. I import these using the automatic header detection and autdetect numerics to set correct data types.

Depending on the process that generates them, some columns may not match between files and feature different headers.

Furthermore, some columns only start getting values added a few rows down into the file. Here's an example:

 ![capture](https://us1.discourse-cdn.com/elastic/original/3X/3/6/361fe020e2bde443c829b3a8fc84bdfe5e8bef5b.png)

I am able to capture partial sets of these files into Elasticsearch, but end up getting many errors such as this one in the process as well:

```
2020-01-17T16:01:19,784][DEBUG][o.e.a.b.TransportShardBulkAction] [AMIENS] [llmlogs-2020.01-000001][0] failed to execute bulk item (index) index {[llmlogs-2020.01-000001][_doc][6wEFtG8B9YgiWpSw2Rkn], source[n/a, actual length: [10.9kb], max length: 2kb]}
org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [WorkingSetSize] of type [float] in document with id '6wEFtG8B9YgiWpSw2Rkn'. Preview of field's value: 'WorkingSetSize'

```

It would appear the automatic column recognition is attempting to recognise the header as a value as well. Do you have any advice on why this may be happening?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 17, 2020, 5:32pm UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493/2 "2020-01-17T17:32:47Z")

</div>

Do you have pipeline.workers set to 1?

---

<div class="post-metadata">

### Author: ![mhyager](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@mhyager](https://discuss.elastic.co/u/mhyager)
#### Post date: [January 20, 2020, 9:40am UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493/3 "2020-01-20T09:40:29Z")

</div>

> [@Badger](#):
>
> pipeline.workers

Yes, I had that set to 1 all throughout development. It was mentioned this was necessary for column header auto detection.

Here's my logstash config:

```
- pipeline.id: pipelineLLM
  pipeline.workers: 1
  pipeline.batch.size: 1
  path.config: "/depot/YAGERTools/ELK/7_4_2/logstash-7_4_2/config/pipelineLLM.config"

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 17, 2020, 9:40am UTC](https://discuss.elastic.co/t/csv-filter-plugin-and-changing-column-headers/215493/4 "2020-02-17T09:40:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
