# BulkIndexError using Elasticsearch in Python
**URL:** https://discuss.elastic.co/t/bulkindexerror-using-elasticsearch-in-python/219417
**Category:** Elasticsearch
**Created:** [February 14, 2020, 4:46pm UTC](https://discuss.elastic.co/t/bulkindexerror-using-elasticsearch-in-python/219417 "2020-02-14T16:46:32Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@brendanwalker](https://discuss.elastic.co/u/brendanwalker)
#### Post date: [February 14, 2020, 4:46pm UTC](https://discuss.elastic.co/t/bulkindexerror-using-elasticsearch-in-python/219417/1 "2020-02-14T16:46:32Z")
I am writing a program to search through really large (\>400mb) csv files provided by the government. Some of these files have over 1,000,000 rows. This is a local program that roughly 5 people will use in my company to help them do their job better. I am using Python (x64) and have tried the native CSV import and Pandas import. Both methods produce the same result. I can read from a small test file (csv) perfectly fine and input them into the ES Index. But when I attempt to input the the large CSV file I get BulkIndexErrors and nothing indexes. What would be the proper to index a large csv file into Elasticsearch using Python?
with open(editContract.get()) as f:
csv\_data = csv.DictReader(f, dialect='excel')
for row in csv\_data:
helpers.bulk(es, row, index="contract\_search", doc\_type='\_doc')
raise BulkIndexError("%i document(s) failed to index." % len(errors), errors)
elasticsearch.helpers.errors.BulkIndexError: ('45 document(s) failed to index...'reason': 'Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes'
---
### Author:  [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [February 14, 2020, 5:05pm UTC](https://discuss.elastic.co/t/bulkindexerror-using-elasticsearch-in-python/219417/2 "2020-02-14T17:05:22Z")
Look at this [issue](https://github.com/elastic/elasticsearch-rails/issues/606), it looks like most of the time the problem was in the format of the data.
Also, you don't have to send bulk a row at a time, I build data in a loop:
```
es_out.append(dict(es_row))
```
Then send it all.
```
spf_index = bulk(client, es_out)
```
Of course, within a reasonable number of rows 🙂
---
### Author:  [@brendanwalker](https://discuss.elastic.co/u/brendanwalker)
#### Post date: [February 14, 2020, 5:30pm UTC](https://discuss.elastic.co/t/bulkindexerror-using-elasticsearch-in-python/219417/3 "2020-02-14T17:30:33Z")
Forgive me as I have little experience with Elasticsearch. However, is there a difference in a Python dictionary and the object you are suggesting by appending rows? Am I supposed to transform the data in some way? I don't fully understand what you are suggesting with the code provided.
Also, I looked at the link you provided. I don't see anything particularly helpful there. The code works perfectly fine for small csv files as it is. It does not work for large csv files with thousands--\>millions of rows. I don't understand why.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 13, 2020, 5:30pm UTC](https://discuss.elastic.co/t/bulkindexerror-using-elasticsearch-in-python/219417/4 "2020-03-13T17:30:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.