# Remove\_field in csv filter section

**URL:** <https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683>\
**Category:** Logstash\
**Created:** [December 20, 2017, 4:52pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683 "2017-12-20T16:52:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amouzigh\_Driss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amouzigh_driss/32/25868_2.png) [@Amouzigh\_Driss](https://discuss.elastic.co/u/Amouzigh_Driss)\
**Post date:** [December 20, 2017, 4:52pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/1 "2017-12-20T16:52:45Z")

</div>

Hello,

I'm trying to remove "extra" fileds while parsing csv files (separator is ; ).  
My filter section is that one:

filter {  
csv {  
separator =\> ";"  
source =\> message  
columns =\> ["ID","Value","Timestamp","DateAction","ServerName"]  
remove\_field = ["message","host","path","@version","@timestamp"]  
}  
}

when remove\_field is commented (#remove\_field) it's working but some extra fields are added : message, host, path, @version, @timestamp  
How car I remove them. When trying I have an error.

Regards

Driss

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [December 20, 2017, 6:44pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/2 "2017-12-20T18:44:15Z")

</div>

Those are default fields that (I think) Logstash adds. I think you (probably?) shouldn't remove them? I'd be interested in learning if this would actually break anything.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 20, 2017, 7:09pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/3 "2017-12-20T19:09:13Z")

</div>

For something like csv's, removing message if the csv parses successfully makes sense, since otherwise you are indexing everything twice. There should be no issue with removing path if you do not care which file a document came from, nor host if you don't care where the file was. I cannot remember what elasticsearch does if the message is missing the @version or @timestamp fields. It may well add them in itself.

```
remove_field = ["message","host","path","@version","@timestamp"]
```

should be

```
remove_field => ["message","host","path","@version","@timestamp"]
```

---

<div class="post-metadata">

**Author:** ![arisbanach](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arisbanach](https://discuss.elastic.co/u/arisbanach)\
**Post date:** [December 20, 2017, 7:12pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/4 "2017-12-20T19:12:46Z")

</div>

Did you try removing the field in a new block after the csv block using mutate?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 20, 2017, 7:34pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/5 "2017-12-20T19:34:18Z")

</div>

@arisbanach That would also work, what I meant was that the original user's problem was that they were missing the \> in the remove\_field directive in their configuration, which is why they got an error if they did not comment it out.

And I tested it, and elasticsearch does not object to documents missing @version and @timestamp. Obviously there is a lot of functionality that requires timestamps, but if Amouzigh\_Driss's use case does not require that functionality I think he will be OK.

---

<div class="post-metadata">

**Author:** ![Amouzigh\_Driss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amouzigh_driss/32/25868_2.png) [@Amouzigh\_Driss](https://discuss.elastic.co/u/Amouzigh_Driss)\
**Post date:** [December 20, 2017, 7:46pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/6 "2017-12-20T19:46:40Z")

</div>

Thank you all for your answers.  
The missing \> was just when I wanted to post the issue.  
I have already timestamp infirnation in csv files. Thus the additional field @timestamp has no added value.  
Will try tomorrow and tell you.  
Best regards  
Driss

---

<div class="post-metadata">

**Author:** ![Amouzigh\_Driss](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amouzigh_driss/32/25868_2.png) [@Amouzigh\_Driss](https://discuss.elastic.co/u/Amouzigh_Driss)\
**Post date:** [December 21, 2017, 12:33pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/7 "2017-12-21T12:33:14Z")

</div>

Yeah. It's working quit fine.Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 18, 2018, 12:33pm UTC](https://discuss.elastic.co/t/remove-field-in-csv-filter-section/112683/8 "2018-01-18T12:33:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.