# How to index CSV files with big number of columns/fields

**URL:** <https://discuss.elastic.co/t/how-to-index-csv-files-with-big-number-of-columns-fields/302555>\
**Category:** Elasticsearch\
**Created:** [April 16, 2022, 3:19pm UTC](https://discuss.elastic.co/t/how-to-index-csv-files-with-big-number-of-columns-fields/302555 "2022-04-16T15:19:19Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ilia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ilia/32/60042_2.png) [@Ilia](https://discuss.elastic.co/u/Ilia)\
**Post date:** [April 16, 2022, 3:19pm UTC](https://discuss.elastic.co/t/how-to-index-csv-files-with-big-number-of-columns-fields/302555/1 "2022-04-16T15:19:19Z")

</div>

Hi all,

We have a lot of csv files with varied number of columns. The majority just have few columns but some have more than 1000 columns. At first we decided to consider each column as a field but the `index.mapping.total_fields.limit` will be violated by some files. We can increase this parameter, of course, but that will be something like the [Sorites paradox](https://en.wikipedia.org/wiki/Sorites_paradox). I mean if we increase this number to 1001 what if a new file has 1002 columns and in general if we increase it to `N` what if a new file has `N+1` columns and so on. This problem is besides the performance issues that potentially will be caused by big number of fields.

I know we can use the [Flattened field type](https://www.elastic.co/guide/en/elasticsearch/reference/current/flattened.html) but I guess the performance will be decreased compared to the column as field mode. So we caught half-way between the two options! I'm curious to know if there is a third option? And in general what is the best practice for such a cases in Elasticseach?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2022, 3:20pm UTC](https://discuss.elastic.co/t/how-to-index-csv-files-with-big-number-of-columns-fields/302555/2 "2022-05-14T15:20:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.