# Python bulk interface changes in 2.0?

**URL:** <https://discuss.elastic.co/t/python-bulk-interface-changes-in-2-0/33408>\
**Category:** Elasticsearch\
**Created:** [October 30, 2015, 6:43pm UTC](https://discuss.elastic.co/t/python-bulk-interface-changes-in-2-0/33408 "2015-10-30T18:43:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![xntang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xntang/32/5606_2.png) [@xntang](https://discuss.elastic.co/u/xntang)\
**Post date:** [October 30, 2015, 6:43pm UTC](https://discuss.elastic.co/t/python-bulk-interface-changes-in-2-0/33408/1 "2015-10-30T18:43:13Z")

</div>

I am using the Python bulk API to insert meta-data into ES. I also parallelized the insertion processing using Python multiprocessing package in which fresh\_interval is set to 30s.

After I changed to ES 2.0, it will report "time out" messge for the first 1,000 elements though I set it 30 seconds and even 60 seconds. Previous ES 1.7.2 doesn't have this problem.

Here is the code snippet, and actuall "size" is set to 1,000. If I understand the API correctly, chunk\_size is the maximal size and the actual number of elements can be smaller than that. Is this correct in ES 2.0?

* * *

def tcp\_worker(ES, sub\_lines, index\_name, size):  
timeout\_t = 30  
tcp\_dump\_list = []  
j = 0  
for line in sub\_lines:  
line\_list = line.split()  
s\_ip = line\_list[2]  
s\_port = int(line\_list[3])  
d\_ip = line\_list[4]  
d\_port = int(line\_list[5])  
bytes = int(line\_list[6])  
packets= int(line\_list[7])  
state = int(line\_list[8])

```
      new_stime = line_list[0] + "T" + line_list[1] + "+08:00"
      a_tcp = {
                  "_index": index_name,
                  "_type" : "tcp_state_type",
                  "_source": {
                        "@ts": new_stime,
                        "sip": s_ip,
                        "sport": s_port,
                        "dip": d_ip,
                        "dport": d_port,
                        "packets": packets,
                        "bytes": bytes,
                        "state": state}
                       }
          tcp_dump_list.append(a_tcp)
          j += 1
          if ( (j % size) == 0) :
              try:
                  helpers.bulk(ES, tcp_dump_list, chunk_size=size, request_timeout=timeout_t)
              except Exception, e:
                  print "%dth chunk, Timeout=%d" %(j, timeout_t)
                  timeout_t = timeout_t + timeout_t
              del tcp_dump_list[0:size]

```

#remaining iterms  
if (len(tcp\_dump\_list)\>0) ：  
try:  
helpers.bulk(ES, tcp\_dump\_list, chunk\_size=size, request\_timeout=timeout\_t)  
except Exception, e:  
print "Error %s Timeout=%d" %(str(e), timeout\_t)

---

<div class="post-metadata">

**Author:** ![xntang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xntang/32/5606_2.png) [@xntang](https://discuss.elastic.co/u/xntang)\
**Post date:** [October 30, 2015, 8:02pm UTC](https://discuss.elastic.co/t/python-bulk-interface-changes-in-2-0/33408/2 "2015-10-30T20:02:15Z")

</div>

I think the question here is: what changes had made in ES 2.0 compared to 1.7.x that affect \_bulk indexing performance?

Which parameters should we tune in ES 2.0 to increase the indexing performance after using the \_bulk API?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:41pm UTC](https://discuss.elastic.co/t/python-bulk-interface-changes-in-2-0/33408/3 "2017-07-05T23:41:35Z")

</div>

Closed on July 5, 2017, 11:41pm UTC