# Error parsing csv

**URL:** <https://discuss.elastic.co/t/error-parsing-csv/314099>\
**Category:** Logstash\
**Created:** [September 10, 2022, 1:33am UTC](https://discuss.elastic.co/t/error-parsing-csv/314099 "2022-09-10T01:33:44Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![manuel.urbano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.urbano/32/110355_2.png) [@manuel.urbano](https://discuss.elastic.co/u/manuel.urbano)\
**Post date:** [September 10, 2022, 1:33am UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/1 "2022-09-10T01:33:44Z")

</div>

Hi, I'm reading a CSV file and sending it to Elasticsearch, everything works fine but some rows are throwing an exception:

```auto
Error parsing csv {:field=>"message", :source=>"6123464a-420f-4838-8ecb-fcce87f16297;208783762198", :exception=>#<ThreadError: interrupted in FiberQueue.pop>}

```

I tried googling and also searching in this forum, but I don't find any results with this exception message.  
What could be causing this exception? After finishing the data ingestion I can see all the results but those shown in the exception are missing.

The conf is:

```auto
input {
  file {
    path => "/usr/share/logstash/files/file.csv"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  csv {
    separator => ";"
    skip_header => true
    skip_empty_rows => true
    columns => ["id", "field1", "field2"]
  }

  mutate {
    copy => { "id" => "[@metadata][_id]"}
    strip => ["field1", "field2"]
    remove_field => ["id", "message", "@timestamp", "path", "host", "@version", "event", "log"]
  }
}

output {
# stdout { codec => rubydebug { metadata => true } }
  elasticsearch {
    ...
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 10, 2022, 5:15am UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/2 "2022-09-10T05:15:00Z")

</div>

What version are you running?

> [@Logstash number of threads keeps increasing until crash](https://discuss.elastic.co/t/logstash-number-of-threads-keeps-increasing-until-crash/313463):
>
> Hi, I'm testing the following pipeline (stack version 8.4.1): input { google\_cloud\_storage {...} } filter {csv {...} mutate {...} } output { elasticsearch {...} } with a large number of events (3M+). After a few minutes, the number of threads increases until more than 18700 and logstash crashes with: [689.902s][warning][os,thread] Failed to start thread "Unknown thread" - pthread\_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. [689.902s][warning][os,thread…

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 10, 2022, 7:52am UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/3 "2022-09-10T07:52:00Z")

</div>

What is a file size?  
Have you tried to increase the memory?  
In config/jvm.options file, set these values to 4-8 GB and don't use rubydebug as you posted.

```auto
-Xms1g
-Xmx1g

```

---

<div class="post-metadata">

**Author:** ![manuel.urbano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.urbano/32/110355_2.png) [@manuel.urbano](https://discuss.elastic.co/u/manuel.urbano)\
**Post date:** [September 10, 2022, 1:02pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/4 "2022-09-10T13:02:29Z")

</div>

@stephenb I'm using 8.4.1, I'll take a look to the link you posted.

I'm running a file with a size of ~500mb, and has ~4.5M rows

---

<div class="post-metadata">

**Author:** ![manuel.urbano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.urbano/32/110355_2.png) [@manuel.urbano](https://discuss.elastic.co/u/manuel.urbano)\
**Post date:** [September 10, 2022, 1:04pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/5 "2022-09-10T13:04:59Z")

</div>

@Rios I'm running ELK with docker, and I have set:

For Elastic Search:  
`ES_JAVA_OPTS: -Xms2G -Xmx2G`

For Logstash:  
`LS_JAVA_OPTS: -Xms1G -Xmx1G`

Also, in my Docker desktop resources I've assigned 10GB

`and don't use rubydebug as you posted.` No, I'm not using it. It's a comment in the file.

EDIT:  
I've increased the `LS_JAVA_OPTS` to 2G in my docker-compose and now is getting worse, I'll paste below part of the breakdown:

```auto

[2022-09-10T13:08:23,483][INFO][logstash.outputs.elasticsearch][my-pipeline] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[153.119s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[153.122s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[153.123s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[153.123s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Ruby-0-Fiber-77912"
[153.124s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[153.124s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Ruby-0-Fiber-77911"
[153.127s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Ruby-0-Fiber-77913"
[153.127s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Ruby-0-Fiber-77910"
[153.127s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[153.127s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Ruby-0-Fiber-77914"
[153.127s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 2048k, guardsize: 0k, detached.
[153.128s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Ruby-0-Fiber-77915"

[2022-09-10T13:10:34,649][FATAL][org.logstash.Logstash][my-pipeline] uncaught error (in thread [my-pipeline]>worker5)

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
at java.lang.Thread.start0(Native Method) ~[?:?]
at java.lang.Thread.start(java/lang/Thread.java:802) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.addWorker(java/util/concurrent/ThreadPoolExecutor.java:945) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(java/util/concurrent/ThreadPoolExecutor.java:1364) ~[?:?]
at org.jruby.ext.fiber.ThreadFiber.createThread(org/jruby/ext/fiber/ThreadFiber.java:284) ~[jruby.jar:?]
at org.jruby.ext.fiber.ThreadFiber.initialize(org/jruby/ext/fiber/ThreadFiber.java:56) ~[jruby.jar:?]
at uri_3a_classloader_3a_.jruby.kernel.enumerator.reset(uri:classloader:/jruby/kernel/enumerator.rb:111) ~[?:?]
at uri_3a_classloader_3a_.jruby.kernel.enumerator.next(uri:classloader:/jruby/kernel/enumerator.rb:93) ~[?:?]
at uri_3a_classloader_3a_.jruby.kernel.enumerator.next(uri:classloader:/jruby/kernel/enumerator.rb:17) ~[?:?]
at usr.share.logstash.vendor.jruby.lib.ruby.stdlib.csv.invokeOther2:next(usr/share/logstash/vendor/jruby/lib/ruby/stdlib//usr/share/logstash/vendor/jruby/lib/ruby/stdlib/csv.rb:1316) ~[?:?]
at usr.share.logstash.vendor.jruby.lib.ruby.stdlib.csv.shift(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/csv.rb:1316) ~[?:?]
at usr.share.logstash.vendor.jruby.lib.ruby.stdlib.csv.invokeOther1:shift(usr/share/logstash/vendor/jruby/lib/ruby/stdlib//usr/share/logstash/vendor/jruby/lib/ruby/stdlib/csv.rb:700) ~[?:?]
at usr.share.logstash.vendor.jruby.lib.ruby.stdlib.csv.parse_line(/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/csv.rb:700) ~[?:?]
at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.logstash_minus_filter_minus_csv_minus_3_dot_1_dot_1.lib.logstash.filters.csv.filter(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-csv-3.1.1/lib/logstash/filters/csv.rb:141) ~[?:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178) ~[?:?]
at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1865) ~[jruby.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175) ~[?:?]
at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(org/logstash/config/ir/compiler/FilterDelegatorExt.java:127) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134) ~[logstash-core.jar:?]
at org.logstash.generated.CompiledDataset1.compute(org/logstash/generated/CompiledDataset1) ~[?:?]
at org.logstash.generated.CompiledDataset2.compute(org/logstash/generated/CompiledDataset2) ~[?:?]
at org.logstash.generated.CompiledDataset3.compute(org/logstash/generated/CompiledDataset3) ~[?:?]
at org.logstash.generated.CompiledDataset4.compute(org/logstash/generated/CompiledDataset4) ~[?:?]
at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:347) ~[logstash-core.jar:?]
at org.logstash.config.ir.CompiledPipeline$CompiledUnorderedExecution.compute(org/logstash/config/ir/CompiledPipeline.java:341) ~[logstash-core.jar:?]
at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:87) ~[logstash-core.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:568) ~[?:?]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:442) ~[jruby.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:306) ~[jruby.jar:?]
at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300) ~[?:?]
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:309) ~[jruby.jar:?]
at java.lang.Thread.run(java/lang/Thread.java:833) [?:?]

[2022-09-10T13:10:34,649][FATAL][org.logstash.Logstash][my-pipeline] uncaught error (in thread [my-pipeline]>worker3)
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
...

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 10, 2022, 1:28pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/6 "2022-09-10T13:28:02Z")

</div>

> I'm running a file with a size of ~500mb

This will require at least 4-8 GB for LS. Can you try the same file with setting:

` LS_JAVA_OPTS: -Xms4G -Xmx4G`

Also try to split file on parts 50-100 MB. Is it possible?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 10, 2022, 1:34pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/7 "2022-09-10T13:34:23Z")

</div>

And ... since your ls.conf has a simple logic, have you think about Filebeat?  
To be honest I haven't tried 500mb size file yet. Without test couldn't clearly recommend FB or LS, but since the logic is simple, would try.

---

<div class="post-metadata">

**Author:** ![manuel.urbano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.urbano/32/110355_2.png) [@manuel.urbano](https://discuss.elastic.co/u/manuel.urbano)\
**Post date:** [September 10, 2022, 1:44pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/8 "2022-09-10T13:44:44Z")

</div>

No, I didn't tried Filebeat, I'll take a look.

I've created other index yesterday for other file, with 4.5M rows and I applied some transformations like adding fields searching in other index and I haven't problems. I don't know why this is happening here.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 10, 2022, 1:58pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/9 "2022-09-10T13:58:18Z")

</div>

> unable to create native thread: possibly out of memory or process/resource limits reached  
> This is reason. Test with 4, 6, 8 GB...

`LS_JAVA_OPTS: -Xms4G -Xmx4G`

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 10, 2022, 2:04pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/10 "2022-09-10T14:04:07Z")

</div>

Did you check the link @stephenb shared? There is a bug in version `8.4.*` that is probably the cause of your issue.

There is a [github issue](https://github.com/elastic/logstash/issues/14498) already.

You will need to wait for an updated version of Logstash to fix this, but there is an workaround.

From your `csv` filter you csv file looks pretty simple and you could use `dissect` to parse it.

If your csv file looks like this:

```auto
id;field1;field2

```

You can use the following dissect filter:

```auto
dissect {
    mapping => {
        "message" => "%{id};%{field1};%{field2}"
    }
}

```

---

<div class="post-metadata">

**Author:** ![manuel.urbano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.urbano/32/110355_2.png) [@manuel.urbano](https://discuss.elastic.co/u/manuel.urbano)\
**Post date:** [September 10, 2022, 2:07pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/11 "2022-09-10T14:07:06Z")

</div>

Thanks @leandrojmp I'm on it.  
I'll try dissect and see how it works for me.  
Thanks!

---

<div class="post-metadata">

**Author:** ![manuel.urbano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.urbano/32/110355_2.png) [@manuel.urbano](https://discuss.elastic.co/u/manuel.urbano)\
**Post date:** [September 10, 2022, 4:06pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/12 "2022-09-10T16:06:35Z")

</div>

> [@leandrojmp](#):
>
> ```auto
> dissect {
> mapping => {
> "message" => "%{id};%{field1};%{field2}"
> }
> }
> 
> ```

Thanks @leandrojmp this worked and is extremelly fast compared with csv.  
I only have to see what's happening with some rows because I'm getting a message:  
`Dissector mapping, field found in event but it was empty`

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 8, 2022, 4:07pm UTC](https://discuss.elastic.co/t/error-parsing-csv/314099/13 "2022-10-08T16:07:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.