# How to handle duplicate column name while parsing csv?

**URL:** <https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684>\
**Category:** Logstash\
**Created:** [June 2, 2015, 5:45am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684 "2015-06-02T05:45:38Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 2, 2015, 5:45am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684/1 "2015-06-02T05:45:38Z")

</div>

First help:

I have log where duplicate field name occurs with same values. while parsing that csv file just want to entertain single field and their values.

e.g. Fields\_A |Fields\_B| Fields\_A| Fields\_C

Just want Fields\_A| Fields\_B| Fields\_C to output elasticsearch.

Second help:

how to assign string for the fields having null values. I want to replace field values with some string if they have null values and output them to elastic search.

Any help is much appreciated.

thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 2, 2015, 6:02am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684/2 "2015-06-02T06:02:26Z")

</div>

> I have log where duplicate field name occurs with same values. while parsing that csv file just want to entertain single field and their values.

> e.g. Fields\_A |Fields\_B| Fields\_A| Fields\_C

> Just want Fields\_A| Fields\_B| Fields\_C to output elasticsearch.

Just delete the field you don't want?

```
filter {
  csv {
    columns => ["fieldA", "fieldB", "fieldA_duplicate", "fieldC"]
    ...
    remove_field => ["fieldA_duplicate"]
  }
}

```

> how to assign string for the fields having null values. I want to replace field values with some string if they have null values and output them to Elasticsearch.

The best I can come up with is a [ruby filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html):

```
ruby {
  code => "
    event.to_hash.each_pair { |k, v|
      event[k] = 'replacement string' if v.nil?
    }
  "
}

```

---

<div class="post-metadata">

**Author:** ![Saket\_Kumar](https://avatars.discourse-cdn.com/v4/letter/s/c57346/32.png) [@Saket\_Kumar](https://discuss.elastic.co/u/Saket_Kumar)\
**Post date:** [June 2, 2015, 6:10am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684/3 "2015-06-02T06:10:11Z")

</div>

thanks magnus for reply....

Just one doubt for the duplicate field deletion:

In my case both field has same header name; hence by deleting as suggested above "remove\_field =\> ["fieldA\_duplicate"]" will not be cause of removal for both the column correct?

regards

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 2, 2015, 6:33am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684/4 "2015-06-02T06:33:19Z")

</div>

But does Logstash even pick up the header names? You're naming them with the `columns` parameter, no? This'll be easier if you show us what an actual message (as parsed by Logstash) looks like.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:38am UTC](https://discuss.elastic.co/t/how-to-handle-duplicate-column-name-while-parsing-csv/1684/5 "2017-07-06T05:38:50Z")

</div>