# CSV files slow input/fitering comparing to python script

**URL:** <https://discuss.elastic.co/t/csv-files-slow-input-fitering-comparing-to-python-script/354886>\
**Category:** Logstash\
**Created:** [March 6, 2024, 10:06pm UTC](https://discuss.elastic.co/t/csv-files-slow-input-fitering-comparing-to-python-script/354886 "2024-03-06T22:06:03Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Benjamin\_Lin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benjamin_lin/32/131971_2.png) [@Benjamin\_Lin](https://discuss.elastic.co/u/Benjamin_Lin)\
**Post date:** [March 6, 2024, 10:06pm UTC](https://discuss.elastic.co/t/csv-files-slow-input-fitering-comparing-to-python-script/354886/1 "2024-03-06T22:06:03Z")

</div>

Hi I have a 4 columns csv data to be proceed by Logstash and aggregating before sending to Elasticsearch. However, from my testing, I tune the JVM heap size up along with batch size to 50k, batch delay 1m. It stil took 1~2 minutes to go through 1.5 million rows in a 48MB file for aggregation. A simple Python script took only 0.5 seconds. Do I do anything wrong here?  
below is my filter config and the input/output is just very simple.

```auto
filter {

 csv {
        separator => ","
        skip_header => "true"
        columns => ["id", "abc-id", "azimuth", "elevation"]
        convert => {
          "id" => "integer"
          "azimuth" => "integer"
          "elevation" => "integer"
    }
  }
  aggregate {
    task_id => "%{id}|%{azimuth}|%{elevation}"
    code => "
        map['count'] ||= 0; map['count'] += 1;
        map['id'] = event.get('id')
        map['azimuth'] = event.get('azimuth')
        map['elevation'] = event.get('elevation')
    "
    push_map_as_event_on_timeout => true
    timeout => 120 # You might want to increase this from 2 seconds
    timeout_code => "
        event.set('[@metadata][wanted]', true)
    "
  }
  if ![@metadata][wanted] { drop {} } # Drop the raw events, just keep the aggregates
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 3, 2024, 10:06pm UTC](https://discuss.elastic.co/t/csv-files-slow-input-fitering-comparing-to-python-script/354886/2 "2024-04-03T22:06:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.