# Speed indexing **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-app-search\ **Created:** [October 24, 2022, 7:25am UTC](https://discuss.elastic.co/t/speed-indexing/317295 "2022-10-24T07:25:30Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Julie\_Montels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julie_montels/32/112379_2.png) [@Julie\_Montels](https://discuss.elastic.co/u/Julie_Montels)\ **Post date:** [October 24, 2022, 7:25am UTC](https://discuss.elastic.co/t/speed-indexing/317295/1 "2022-10-24T07:25:30Z")
Hello, When pushing our data to Elastic (we made sequential batches of 100), it takes a lot of time even for a small amount of data (24h for 1.5Million documents). Is it normal ? What do you recommend to speed the indexing ? Thanks ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 24, 2022, 10:56am UTC](https://discuss.elastic.co/t/speed-indexing/317295/2 "2022-10-24T10:56:00Z")
Welcome! Just to make sure: - Are you sending the data to AppSearch or Elasticsearch? - How are you sending the data? Which API are you calling? - What is the source of the data? How are you fetching the data from the source? - Where are you running Elastic? On Elastic cloud? On a local machine? - Which hardware do you have? ---
**Author:** ![Julie\_Montels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julie_montels/32/112379_2.png) [@Julie\_Montels](https://discuss.elastic.co/u/Julie_Montels)\ **Post date:** [October 24, 2022, 12:41pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/3 "2022-10-24T12:41:52Z")
- App search (so batch of 100 max) - I'm using Astronomer (scheduler) that creates a pod that is launching a python script. The client is elastic-enterprise-search (8.4) - The script is the following : ```auto client = AppSearch(self.host, http_auth=(self.user, self.password), request_timeout=6000, max_retries=10, retry_on_timeout=True) for upsert_place in range(0, len(docs), 100): responses = client.index_documents( engine_name=engine, documents=docs[upsert_place : upsert_place + self.upsert_size], ) ``` - so `docs` is already a list of JSON, **we take all this data from Snowflake where we have 1 table per engine** - Indexing using Astronomer takes at least 2/3 secs per batch of 100, from my PC it's around 5 secs per batch ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 24, 2022, 12:57pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/4 "2022-10-24T12:57:12Z")
How long does it take to "just" read the content from Snowflake? ---
**Author:** ![Julie\_Montels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julie_montels/32/112379_2.png) [@Julie\_Montels](https://discuss.elastic.co/u/Julie_Montels)\ **Post date:** [October 24, 2022, 12:59pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/5 "2022-10-24T12:59:03Z")
between 5 to 10 seconds I would say ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 24, 2022, 1:11pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/6 "2022-10-24T13:11:12Z")
For the whole dataset you mean? Could you measure that? ---
**Author:** ![Julie\_Montels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julie_montels/32/112379_2.png) [@Julie\_Montels](https://discuss.elastic.co/u/Julie_Montels)\ **Post date:** [October 24, 2022, 1:19pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/7 "2022-10-24T13:19:02Z")
For instance, when using my PC, it takes 5 secs - 10 secs to retrieve 2Million rows as a list of JSON from Snowflake (docs that correspond to 1 engine) -\> and to push them, I split them into batches of 100 and it takes 2 or 5 seconds per batch. So for 2Million rows, it takes days! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [October 25, 2022, 8:59am UTC](https://discuss.elastic.co/t/speed-indexing/317295/8 "2022-10-25T08:59:52Z")
Just out of curiosity, could you send the data to Elasticsearch instead of AppSearch with batches of 10000 for example and tell how fast it is? What does a typical document look like? ---
**Author:** ![marone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marone/32/87145_2.png) [@marone](https://discuss.elastic.co/u/marone)\ **Post date:** [October 25, 2022, 10:15am UTC](https://discuss.elastic.co/t/speed-indexing/317295/9 "2022-10-25T10:15:43Z")
From what I understood from your code snippet and since you are indexing using batches, you don't need to loop over the list `docs` and index each element. Based on [this example](https://www.elastic.co/guide/en/enterprise-search-clients/python/current/app-search-api.html#_create_and_index_documents), the parameter `documents` accepts a list of json object, which is your case for `docs`, so for me, I'd try this instead: ```auto responses = client.index_documents( engine_name=engine, documents=docs, ) ``` I didn't use AppSearch before and don't know why the max is 100 docs / batch, why not more? But as @dadoonet pointed you can try sending data to elasticsearch and use the [bulk](https://elasticsearch-py.readthedocs.io/en/v8.4.3/api.html#elasticsearch.Elasticsearch.bulk) function. `res = es.bulk(docs, index="my-index")` Then test for many batch size: 100, 1000, 10000, etc. and see how your cluster will perform while indexing! Hope it helps 🙂 ---
**Author:** ![Julie\_Montels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/julie_montels/32/112379_2.png) [@Julie\_Montels](https://discuss.elastic.co/u/Julie_Montels)\ **Post date:** [October 25, 2022, 2:03pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/10 "2022-10-25T14:03:18Z")
We can't index more than 100 documents at a single time using app search, that's why I'm looping on docs to take docs 100 per 100 - take a subset of the list then ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 22, 2022, 2:04pm UTC](https://discuss.elastic.co/t/speed-indexing/317295/11 "2022-11-22T14:04:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.