# What is the most efficient way to insert data to Elasticsearch?

**URL:** <https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647>\
**Category:** Elasticsearch\
**Created:** [June 20, 2018, 9:14am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647 "2018-06-20T09:14:47Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [June 20, 2018, 9:14am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/1 "2018-06-20T09:14:47Z")

</div>

I try to use [Python Elasticsearch Client](https://elasticsearch-py.readthedocs.io/en/master/) for inserting data to Elasticsearch however performance is terrible. I use Ubuntu 14 with 164 GB RAM and 40 processors Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz. I would like to achieve efficiency equals or close to 100000 records per second however at the moment after running this source code:

```
from elasticsearch import Elasticsearch

es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)

print("time 0: " + str(datetime.now()))

for key in range(1000):

    es.index(index='messages', doc_type='message', body={
        'message': "example message",
    })

print("time 1: " + str(datetime.now()))

```

I get result:

```
time 0: 2018-06-20 10:38:08.311971
time 1: 2018-06-20 10:38:36.154774

```

I tried also version with bulk in this way:

```
from elasticsearch import Elasticsearch
from elasticsearch import helpers

es = Elasticsearch([{'host': 'localhost', 'port': 9200}])
es = Elasticsearch(timeout=30, max_retries=10, retry_on_timeout=True)

actions = []
for key in range(1000):

    actions.append(
      {
        "_index": "messages",
        "_type": "message",
        "_source": {
            "message": "example message"}
      }
    )
print("time 0: " + str(datetime.now()))
helpers.bulk(es, actions)
print("time 1: " + str(datetime.now()))

```

but the result is still very bad:

```
time 0: 2018-06-20 10:51:25.667748
time 1: 2018-06-20 10:51:29.757604

```

Any ideas how can I improve this?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 20, 2018, 10:10am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/2 "2018-06-20T10:10:30Z")

</div>

What type of disk do you have? How many concurrent indexing threads/processes are you using? Which version of Elasticsearch are you using? Also have a look at [this guide](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/tune-for-indexing-speed.html).

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [June 20, 2018, 10:57am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/3 "2018-06-20T10:57:38Z")

</div>

`cat /sys/block/sda/queue/rotational` returns `0` what means SSD. At the moment I run only one thread. `curl -XGET 'localhost:9200'` returns `"version" : { "number" : "6.2.4", ...` Any ideas how to improve performance?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 20, 2018, 11:00am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/4 "2018-06-20T11:00:15Z")

</div>

Use multiple indexing threads. You will not be able to saturate the node using a single connection. If your test records are very small you may also want to test with a larger bulk size.

---

<div class="post-metadata">

**Author:** ![white\_rabbit](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/white_rabbit/32/46888_2.png) [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\
**Post date:** [June 20, 2018, 11:25am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/5 "2018-06-20T11:25:34Z")

</div>

I test different bulk sizes. Can you recommend example source code with correct use of multithreading with Elasticsearch in Python?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 20, 2018, 11:52am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/6 "2018-06-20T11:52:20Z")

</div>

When using Python it may be better to divide the work between multiple processes.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2018, 11:52am UTC](https://discuss.elastic.co/t/what-is-the-most-efficient-way-to-insert-data-to-elasticsearch/136647/7 "2018-07-18T11:52:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.