# Logstash Ruby Parse CSV and Validate Its Fields **URL:** \ **Category:** Logstash\ **Created:** [September 7, 2017, 12:13am UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643 "2017-09-07T00:13:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Robert.S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert.s/32/22039_2.png) [@Robert.S](https://discuss.elastic.co/u/Robert.S)\ **Post date:** [September 7, 2017, 12:13am UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/1 "2017-09-07T00:13:07Z")
Hello, I am trying to monitor a directory, parse csv files, and validate some of the fields' values. I thought I could do the following but failed. Any suggestion? I am open for alternatives... input { file { path =\> "C:/test/\*.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," #my fields columns =\> ["field1","field2","field3","field4"] } ruby{ code =\> “ #my array for validation ary = [“value1”,”value2”,”value3”] if ary.include? (event["field2"]) #create new field, assign invalid as value event["field5"] = "Invalid" else #create new field, assign valid as value event["field5"] = "Valid" end “ } } output { elasticsearch { hosts =\> "[http://localhost:9200](http://localhost:9200)" index =\> "test\_index" } stdout {} } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 7, 2017, 7:25am UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/2 "2017-09-07T07:25:05Z")
For best results when asking a question, always quote any error messages or give other details that points at the problem. Your problem is most likely that you're nesting double quotes inside double quotes in your ruby filter. Do: ``` code => " ... ' ... " code => ' ... " ... ' ``` Do not: ``` code => " ... " ... " code => ' ... ' ... ' ``` ---
**Author:** ![Robert.S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert.s/32/22039_2.png) [@Robert.S](https://discuss.elastic.co/u/Robert.S)\ **Post date:** [September 8, 2017, 6:28pm UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/3 "2017-09-08T18:28:59Z")
Hi Magnus, Sorry I wasn't clear. I am trying to validate a field value during parsing/posting to elastic/kibana. I am not sure what is the best way to achieve that, my end goal is to validate all the data I am posting, some fields has a list of valid values, some are dates only, others are integer, etc... My first attempt was the field with a list of valid values (shown below success) -\> it would be better if I know how to define an array to validate against it. Next, I need to check field value formatting, example: [field1] should be formatted =\> '0123-45A-678' below is what I found easier than using Ruby: if ([field\_to\_validate] in ['value1', 'value2', 'value3']){ mutate{add\_field =\> { "validation" =\> "invalid" }} } else{ mutate{add\_field =\> { "validation" =\> "valid" }} } ---
**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\ **Post date:** [September 8, 2017, 7:19pm UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/4 "2017-09-08T19:19:55Z")
Have a look at the `translate` filter. It offers a lookup solution. You may need to use two translate filters with an if block to exclude invalid values from a regex check. set the `exact` config option to `true` for both. **Value Validation** It allows you to define a list of invalid -\> flag pairs - match against the first and add the second as a value to the `@target` field. e.g. ```auto "foo", "invalid", "bar", "invalid" ``` **Format Validation** set the `regex` config option to `true`. It allows you to define a list of regex -\> flag pairs - match the regex and add the flag to the `@target`. e.g. ```auto "^\d{4}-\d{2}[A-z]-\d{3}$", "format valid" ``` The dictionaries can be loaded from a file with periodic refresh if you want to change them. Use this site to build regexes: [http://rubular.com/](http://rubular.com/) ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 11, 2017, 6:04am UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/5 "2017-09-11T06:04:55Z")
> Sorry I wasn't clear. I am trying to validate a field value during parsing/posting to elastic/kibana. No, that certainly wasn't clear but the configuration you posted nevertheless has a problem with nested quoting. ---
**Author:** ![Robert.S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert.s/32/22039_2.png) [@Robert.S](https://discuss.elastic.co/u/Robert.S)\ **Post date:** [September 12, 2017, 6:44pm UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/6 "2017-09-12T18:44:20Z")
You are correct! This was just an example of what I am trying to achieve. I was able to fix quoting error. As I mentioned, I am new at this, your help is highly appreciated... Now I am trying to figure out how to enable/plugin my regex to validate field's formats and values... ---
**Author:** ![Robert.S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/robert.s/32/22039_2.png) [@Robert.S](https://discuss.elastic.co/u/Robert.S)\ **Post date:** [September 12, 2017, 6:46pm UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/7 "2017-09-12T18:46:31Z")
Thank you! I think this is very close to what I am looking for. I just need to read a little more on how to plugin the regex. I cannot find a sample code where regex - exact = true, and whether I need to add ruby filter for that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 10, 2017, 6:46pm UTC](https://discuss.elastic.co/t/logstash-ruby-parse-csv-and-validate-its-fields/99643/8 "2017-10-10T18:46:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.