# Logstash csv imports not all data into elastic search

**URL:** <https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199>\
**Category:** Logstash\
**Created:** [April 3, 2023, 10:12am UTC](https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199 "2023-04-03T10:12:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 3, 2023, 10:12am UTC](https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199/1 "2023-04-03T10:12:55Z")

</div>

Hi guys!  
Do you have any idea why I can't import all the data from csv.file via logstash into Elasticsearch? it always imports the same amount equal to 6873 and should be more than 53k. at runtime, the console displays the imported data but does not get into Elasticsearch. I don't get any errors.

Data  
example of one row from csv.file

```auto
SKU;Kategoria;Rodzina;Marka;Kod u dostawcy;Kod u dostawcy 2;Nazwa towaru
00000290;1_brands,1_restaurant_equipment,2_brand_rm_gastro,2_food_holding_and_warming_equipment,3_steam_heaters_and_buffets,4_bain_marie_heaters,categoryE7E7163;bemary grzewcze;rm gastro;00000290;BMPD 2120;Bemar elektryczny 2-komorowy, GN 1/1, jezdny z kranem spustowym, 1,4 kW;

```

code

```auto
input{
  file {
	path => "C:/db.csv" 
    start_position => "beginning"
    sincedb_path => "NULL"
  }	  
}

filter {
   csv { 
	separator => ";"
        skip_empty_columns => true
	columns => ["SKU","Kategoria","Rodzina","Marka","Kod u dostawcy","Kod u dostawcy2", "Nazwa towaru"]
	}
	
	ruby {
        code => '
            c = event.get("Kategoria")
            if c
                a = c.split(",")
                event.set("Kategoria", a.shift)
                a.each { |x| event.set(x.strip, x.strip) }
            end
        '
    }
}
output{
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "products"
	user => "elastic"
    password => "**" 
  }
stdout{}
}

```

when I comment this part of code I'm able to import all text from csv:

```auto
	ruby {
        code => '
            c = event.get("Kategoria")
            if c
                a = c.split(",")
                event.set("Kategoria", a.shift)
                a.each { |x| event.set(x.strip, x.strip) }
            end
        '
    }

```

But this part of the code works correctly and does not cause problems if I import 1000 lines into elastic, then all the data are stored fine

---

<div class="post-metadata">

**Author:** ![oleksiiorel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiiorel/32/118205_2.png) [@oleksiiorel](https://discuss.elastic.co/u/oleksiiorel)\
**Post date:** [April 4, 2023, 3:50pm UTC](https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199/2 "2023-04-04T15:50:36Z")

</div>

That solved! just specified this setting before or after creating an index. in my case 60000 is enough by default the limit is 1000

```auto
PUT test_index/_settings
{
  "index.mapping.total_fields.limit": 60000
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 2, 2023, 3:51pm UTC](https://discuss.elastic.co/t/logstash-csv-imports-not-all-data-into-elastic-search/329199/3 "2023-05-02T15:51:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.