# CSV parse/read error when dumped into a folder

**URL:** <https://discuss.elastic.co/t/csv-parse-read-error-when-dumped-into-a-folder/196185>\
**Category:** Logstash\
**Created:** [August 21, 2019, 7:33pm UTC](https://discuss.elastic.co/t/csv-parse-read-error-when-dumped-into-a-folder/196185 "2019-08-21T19:33:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![baselai](https://avatars.discourse-cdn.com/v4/letter/b/ea666f/32.png) [@baselai](https://discuss.elastic.co/u/baselai)\
**Post date:** [August 21, 2019, 7:33pm UTC](https://discuss.elastic.co/t/csv-parse-read-error-when-dumped-into-a-folder/196185/1 "2019-08-21T19:33:32Z")

</div>

I have the following case: I configured logstash to monitor a specific root and subdirectories if there is any folder created with a csv file in it then it should read it and push it to elasticsearch.

Here is how I configured it:

```auto
input {
  file {
    path => "/usr/share/input/**/*.csv"
    start_position => beginning
    sincedb_path => "/dev/null"
    discover_interval => 5
    stat_interval => "1 s"
  }
}
filter {
    csv { 
        source => "message" 
        target => "[@metadata][row]"
        autodetect_column_names => true
    }
    ruby {
        code => '
            r = event.get("[@metadata][row]")
            rn = r["row_number"]
            if rn and r
                a = []
                r.each { |k, v|
                    h = {}
                    h["column_name"] = k
                    h["row_number"] = rn.to_i
                    h["column_value_float"] = v
                    h["column_value_string"] = v
                    a << h                    
                }
                event.set("foo", a)
            end
        '
    }
    split { field => "foo" }
    ruby {
        code => '
            event.get("foo").each { |k, v|
                event.set(k,v)
            }
            event.remove("foo")
        '
    }
}
output {
    stdout { codec => rubydebug }
    file {
        path => "/usr/share/output/output.json"
        codec => "json_lines"
    }
}

```

**But** , **it works only** in this use-case:

> - Logstash service is down
> - Created a folder and dumper the csv file in it
> - Run logstash service again
> - Then it will read the file properly and it will read any file in any newly created folder!!

and it **doesn't work** in this use-case:

> - Logstash is up
> - Dump a folder with a csv file in it in the root the logstash is listening to.
> - it will throw an error and crash

Logs output:

```auto
[2019-08-21T19:48:39,973][WARN][logstash.filters.split] Only String and Array types are splittable. field:foo is of type = NilClass
[2019-08-21T19:48:39,977][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClass
[2019-08-21T19:48:39,978][WARN][logstash.filters.split] Only String and Array types are splittable. field:foo is of type = NilClass
[2019-08-21T19:48:39,977][WARN][logstash.filters.split] Only String and Array types are splittable. field:foo is of type = NilClass
[2019-08-21T19:48:39,981][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClass
[2019-08-21T19:48:39,982][ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `each' for nil:NilClass

....
{
    "@timestamp" => 2019-08-21T19:23:09.190Z,
          "tags" => [
        [0] "_split_type_failure",
        [1] "_rubyexception"
    ],
          "path" => "/usr/share/input/sentiment-1/sample.csv",
      "@version" => "1",
          "host" => "b61d8fdbca5e",
       "message" => "1,Mike,Hello,11.5\r"
}
...

```

any ideas @Badger ?

**Update:**

I changed the time from

```auto
discover_interval => 5
stat_interval => "1 s"

```

to

```auto
discover_interval => 3
stat_interval => "2 s"

```

and waited after the service is up, like around 5 secs and I dumped the folders in the root and it worked, any explanation?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 18, 2019, 7:33pm UTC](https://discuss.elastic.co/t/csv-parse-read-error-when-dumped-into-a-folder/196185/2 "2019-09-18T19:33:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.