# How to use parallel\_bulk function

**URL:** <https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741>\
**Category:** Elasticsearch\
**Created:** [May 20, 2019, 6:15am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741 "2019-05-20T06:15:18Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\
**Post date:** [May 20, 2019, 6:15am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/1 "2019-05-20T06:15:19Z")

</div>

I have a document to be indexed to elastic search , it is 200 MB file ,so i want to use parallel bulk..  
file is in this format.  
`[{},{},{},{}]`  
basically it is an array of objects.

but when i try to index using parallel bulk, nothing is being indexed to elastic search.  
how do i index data using parallel bulk?  
Do i need to format the data in any format before i use parallel bulk, if yes please specify the format.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 20, 2019, 6:20am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/2 "2019-05-20T06:20:28Z")

</div>

The bulk interface is for sending multiple documents (not necessarily all) in a single request and [has to follow the format described in the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/docs-bulk.html). It is recommended that bulk requests are limited to around 5MB in size so you should break your data up into multiple requests.

---

<div class="post-metadata">

**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\
**Post date:** [May 20, 2019, 6:23am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/3 "2019-05-20T06:23:36Z")

</div>

Using this format bulk api works but not parallel\_bulk or streaming\_bulk, basically i want to bulk index data in chunks of desired size.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 20, 2019, 6:25am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/4 "2019-05-20T06:25:15Z")

</div>

> [@MKU](#):
>
> parallel\_bulk or streaming\_bulk

What do you mean with this? How are you indexing the data?

---

<div class="post-metadata">

**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\
**Post date:** [May 20, 2019, 6:26am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/5 "2019-05-20T06:26:59Z")

</div>

please check out parallel and streaming bulk section here [https://elasticsearch-py.readthedocs.io/en/master/helpers.html](https://elasticsearch-py.readthedocs.io/en/master/helpers.html)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 20, 2019, 6:28am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/6 "2019-05-20T06:28:24Z")

</div>

It would have helped if you explained that you are using the python client in the forst post. Can you show us your code? Are you parsing the input file and treating the objects one by one in the code?

---

<div class="post-metadata">

**Author:** ![MKU](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mku/32/48327_2.png) [@MKU](https://discuss.elastic.co/u/MKU)\
**Post date:** [May 20, 2019, 7:46am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/7 "2019-05-20T07:46:32Z")

</div>

```
import json

from elasticsearch import Elasticsearch

from elasticsearch.helpers import bulk

from elasticsearch.helpers import parallel_bulk

from elasticsearch.helpers import streaming_bulk

from requests.auth import AuthBase

import requests

import json

requests.packages.urllib3.disable_warnings()

es = Elasticsearch('https://localhost:9200',ca_certs=False,verify_certs=False,headers={'content-type': 'application/x-ndjson','Authorization' :'Bearer P/N1UniTZQ~'})

parallel_bulk(client=es,actions="ldif2.json",index="user_volvo",chunk_size=2)

```

This is the python code that i used.

the data ldif2.json is in this format.

**[{"\_index": "abc", "\_type": "user", "\_source": {"dn": " cn=abc,o=VCC", "changetype": " add", "mail": " [abc@corp.com](mailto:abc@corp.com)", "surname": " satya2"}} ,{},{},{}.......]**

please refer this stackoverflow link,  
[https://stackoverflow.com/questions/54962685/how-to-improve-parallel-bulk-from-python-code-for-elastic-insert](https://stackoverflow.com/questions/54962685/how-to-improve-parallel-bulk-from-python-code-for-elastic-insert)  
[https://stackoverflow.com/questions/54212958/elasticsearch-python-parallel-bulk-can-not-insert-data](https://stackoverflow.com/questions/54212958/elasticsearch-python-parallel-bulk-can-not-insert-data)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 17, 2019, 7:46am UTC](https://discuss.elastic.co/t/how-to-use-parallel-bulk-function/181741/8 "2019-06-17T07:46:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.