# How to restrict fields from csv file using logstash? **URL:** \ **Category:** Logstash\ **Created:** [October 21, 2015, 7:37am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643 "2015-10-21T07:37:16Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![shankar.T](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@shankar.T](https://discuss.elastic.co/u/shankar.T)\ **Post date:** [October 21, 2015, 7:37am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/1 "2015-10-21T07:37:16Z")
For example in csv file i have 100 fields. But i want to load only 10 specific fileds which i needed. How can i achieve this using logstash? Thanks in advance.. Thanks & Regards, Shankarananth.T ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 7:47am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/2 "2015-10-21T07:47:48Z")
The csv filter will always extract all fields, but you can follow up with a ruby filter to delete all fields except the ones you specify. See [http://stackoverflow.com/a/30343349/414355](http://stackoverflow.com/a/30343349/414355). ---
**Author:** ![shankar.T](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@shankar.T](https://discuss.elastic.co/u/shankar.T)\ **Post date:** [October 21, 2015, 10:54am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/3 "2015-10-21T10:54:02Z")
Hai magnusbaeck, First of all Thank you very much for your reply. As you mentioned in the link, I'm trying to pull the fields which are needed for me from csv. For that i'm using the below mentioned filter format in logstash.conf file. input { file { path =\> "D:\logstatsh\logstash-1.5.4\weblog.csv" start\_position =\> "beginning" } } filter { ruby { code =\> " wanted\_fields = ['ApplicationName', 'BrokerName','CounterId'] event.to\_hash.keys.each { |k| event.remove(k) unless wanted\_fields.include? k } " } } output { elasticsearch { action =\> "index" protocol =\> "http" host =\> "localhost" index =\> "Web\_log\_loading" ``` } stdout {} ``` } but the problem is after calling the above mentioned logstash.conf file command prompt data is not getting loaded into elastic search. As i'm very new to ELK i'm not able to get what is the real problem. Thanks and Regards, Shankarananth.T ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 11:04am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/4 "2015-10-21T11:04:36Z")
Logstash is probably tailing weblog.csv and waiting for additional lines to be added to the file. To read the file from the beginning you need to clear the sincedb file. See the file input documentation and countless other questions that talk about this. ---
**Author:** ![shankar.T](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@shankar.T](https://discuss.elastic.co/u/shankar.T)\ **Post date:** [October 21, 2015, 11:10am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/5 "2015-10-21T11:10:44Z")
Hai magnus, Ya sure i will go through it. But for the same below mentioned filter is loading the data completely in elastic search. for that also i'm using the same path and start\_position . if u have some samples for restricting the fields from csv can u post it here. I am a newbie, if i asked anything wrong im sorry. kindly correct me. input { file { path =\> "D:\logstatsh\logstash-1.5.4\dataset.csv" start\_position =\> "beginning" } } filter { csv { separator =\> "," columns =\> ["Date","Open","High","Low","Close","Volume","Adj Close"] } mutate {convert =\> ["High", "float"]} mutate {convert =\> ["Open", "float"]} mutate {convert =\> ["Low", "float"]} mutate {convert =\> ["Close", "float"]} mutate {convert =\> ["Volume", "float"]} } output { elasticsearch { action =\> "index" protocol =\> "http" host =\> "localhost" index =\> "data" ``` } stdout {} ``` } regards, Shankarananth.T ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 11:15am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/6 "2015-10-21T11:15:56Z")
Have you ever before tried to have Logstash read dataset.csv? If not, i.e. the file was previously unseen, the file will be read from the beginning (thanks to `start_position => beginning`). ---
**Author:** ![shankar.T](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@shankar.T](https://discuss.elastic.co/u/shankar.T)\ **Post date:** [October 21, 2015, 11:19am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/7 "2015-10-21T11:19:36Z")
No i never used Logstash read dataset.csv . But will this may be issue in restricting the fileds from csv file. If u have any sample file using ruby filter to restric the field from csv file can u post it here. Thank u ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 21, 2015, 11:23am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/8 "2015-10-21T11:23:17Z")
> No i never used Logstash read dataset.csv Okay. But you've attempted to process weblog.csv more than once, yes? > If u have any sample file using ruby filter to restric the field from csv file can u post it here. I've already posted a StackOverflow link containing an example of this. I have nothing more to add on that matter. ---
**Author:** ![shankar.T](https://avatars.discourse-cdn.com/v4/letter/s/59ef9b/32.png) [@shankar.T](https://discuss.elastic.co/u/shankar.T)\ **Post date:** [October 21, 2015, 12:01pm UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/9 "2015-10-21T12:01:06Z")
ok thank u very much magnus. Let me try again . ---
**Author:** ![kevalbhatt](https://avatars.discourse-cdn.com/v4/letter/k/73ab20/32.png) [@kevalbhatt](https://discuss.elastic.co/u/kevalbhatt)\ **Post date:** [August 9, 2016, 12:09pm UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/10 "2016-08-09T12:09:18Z")
Hi @magnusbaeck as you said I tried ruby example but it gives me error This is my .conf file. `input { file { path => "/home/keval/Keval/NODE/comparison/flipkart_mobile.csv" type => "flipkart" start_position => "beginning" } } filter { ruby { code => " wanted_fields = ['productId', 'title', 'mrp', 'sellingPrice', 'productUrl', 'productFamily'] event.to_hash.keys.each { |k| event.remove(k) unless wanted_fields.include? k } " } } output { elasticsearch { action => "index" hosts => "127.0.0.1:9200" index => "flipkart" workers => 1 document_id => "%{productId}" } stdout {} }` Error: **NoMethodError: undefined method `to\_iso8601' for nil:NilClass** I think its @time field issue. can you please help me to restrict csv fields ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 9, 2016, 12:23pm UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/11 "2016-08-09T12:23:33Z")
Without more context it's impossible to tell what's going wrong. Please try the prune filter instead. It's specifically made for this task. ---
**Author:** ![karthikeyan95](https://avatars.discourse-cdn.com/v4/letter/k/f14d63/32.png) [@karthikeyan95](https://discuss.elastic.co/u/karthikeyan95)\ **Post date:** [August 30, 2016, 9:58am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/12 "2016-08-30T09:58:01Z")
Hi everyone, I need a solution for a similar problem. i have a logstash configuration using the csv filter containing columns say `csv{ columns => ["A", "B", "C","D","E"] seperator => "," skip_empty_columns => true }` and say if my CSV file has values only for [B,C,D,E] ie, there is no column named "A". when i run my Logstash with particular configuration, what happens is that Logstash puts the value of column "B" in place of Column "A" and so on. Finally there is no value for column "E". Since i have given the option of `skip_empty_columns => true` Column "E" is omitted. I am looking for a solution where Logstash has to take values and map them to the respective Column and not in some order. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 30, 2016, 11:31am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/13 "2016-08-30T11:31:47Z")
@karthikeyan95, please start a new thread for your question. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:40am UTC](https://discuss.elastic.co/t/how-to-restrict-fields-from-csv-file-using-logstash/32643/14 "2017-07-06T04:40:55Z")