# How to import 10 millions of documents (about 5G) into app search? **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-app-search\ **Created:** [December 12, 2019, 7:58pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716 "2019-12-12T19:58:02Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![viphuangwei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/viphuangwei/32/59306_2.png) [@viphuangwei](https://discuss.elastic.co/u/viphuangwei)\ **Post date:** [December 12, 2019, 7:58pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/1 "2019-12-12T19:58:02Z")
Hi There, How to import 10 million of documents (about 5G) into the App search (self-managed)? What's the best practice to do it? Is there any limitation to do this. Thanks, Wei. ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [December 13, 2019, 11:55am UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/2 "2019-12-13T11:55:44Z")
All of our limits are noted here: [https://swiftype.com/documentation/app-search/limitations#query-limits](https://swiftype.com/documentation/app-search/limitations#query-limits). I think these all apply to Self-Managed as well, currently, though we are working to lift some of them. I don't believe we have any mechanisms for bulk importing that many documents. You'll probably need to script something against our Indexing API: [https://swiftype.com/documentation/app-search/api/documents#create](https://swiftype.com/documentation/app-search/api/documents#create). Any thoughts here @qhoxie @nickchow? ---
**Author:** ![viphuangwei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/viphuangwei/32/59306_2.png) [@viphuangwei](https://discuss.elastic.co/u/viphuangwei)\ **Post date:** [December 13, 2019, 6:40pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/3 "2019-12-13T18:40:24Z")
Thank you for replying. I think that would be a nice feature that can bulk importing many documents especially help for the fist/full data importing/migration. ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [December 13, 2019, 8:21pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/4 "2019-12-13T20:21:38Z")
I agree @viphuangwei ---
**Author:** ![nickchow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nickchow/32/42547_2.png) [@nickchow](https://discuss.elastic.co/u/nickchow)\ **Post date:** [December 14, 2019, 4:20pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/5 "2019-12-14T16:20:23Z")
Thanks for your comment @viphuangwei. Normally folks implement their own tooling for bulk importing, but we'll keep your feedback in mind. Our client libraries might help, if you haven't seen those already: [https://swiftype.com/documentation/app-search/getting-started#build](https://swiftype.com/documentation/app-search/getting-started#build) ---
**Author:** ![werewolf\_ninja](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@werewolf\_ninja](https://discuss.elastic.co/u/werewolf_ninja)\ **Post date:** [December 29, 2019, 8:58pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/6 "2019-12-29T20:58:52Z")
I have implemented my own script to upload 10K records, but now getting "Rate limit exceeded" error. So how do we by pass this limitation? Using the **[app-search-python](https://github.com/elastic/app-search-python)**. Thanks. ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [January 2, 2020, 12:23pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/7 "2020-01-02T12:23:18Z")
@werewolf_ninja Unfortunately I believe the only solution to this currently is to slow down your ingestion script. ---
**Author:** ![werewolf\_ninja](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@werewolf\_ninja](https://discuss.elastic.co/u/werewolf_ninja)\ **Post date:** [January 3, 2020, 3:32am UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/8 "2020-01-03T03:32:22Z")
Yes, that is what I ended up doing. I think this may differ from one account to another, so far was able to make 5 consecutive api calls every three minutes. In this first attempt, uploaded 5454 records in 31 minutes. At this point of time, the UI upload feature can handle up to 5000 records, so this api rate limitation is very limiting and painful. @JasonStoltz Thank you for your prompt reply, I hope you can bring this issue to the team, and hopefully it gets prioritized 🙂 ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [January 3, 2020, 1:52pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/9 "2020-01-03T13:52:36Z")
@werewolf_ninja I will see if we can get this prioritized, thanks for your perseverance! ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [January 3, 2020, 2:05pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/10 "2020-01-03T14:05:09Z")
@werewolf_ninja Are you using self managed App Search? ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [January 3, 2020, 5:58pm UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/11 "2020-01-03T17:58:52Z")
@werewolf_ninja FWIW, you should be getting much better throughput than 5000 documents every 30 minutes. From what I can see, we should be allowing up to 3000 documents per minute, assuming you are using a paid App Search account. You can use the following headers in the response for reference: ```auto X-RateLimit-Limit → X-RateLimit-Remaining → ``` If you are rate limited and receive a 429 response, you can check the `Retry-After` header which will tell you how long to wait before performing a new request. ---
**Author:** ![werewolf\_ninja](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@werewolf\_ninja](https://discuss.elastic.co/u/werewolf_ninja)\ **Post date:** [January 6, 2020, 1:11am UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/12 "2020-01-06T01:11:38Z")
As far as the self managed App Search, no, but I am still on the trial version. I do appreciate your response about the X-Rate-Limit, that is very helpful. Will also try configuring my script for 3000 docs per each minutes, to see if I would still get a rate limit error. I will post my results here sometime this week. 👍 ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [January 6, 2020, 11:52am UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/13 "2020-01-06T11:52:55Z")
@werewolf_ninja Great. Also, free trials are rate limited to 800 docs per minute, not 3000. ---
**Author:** ![werewolf\_ninja](https://avatars.discourse-cdn.com/v4/letter/w/ac91a4/32.png) [@werewolf\_ninja](https://discuss.elastic.co/u/werewolf_ninja)\ **Post date:** [January 8, 2020, 3:50am UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/14 "2020-01-08T03:50:14Z")
@JasonStoltz Thank you for the clarification about the limitation for trial accounts. That explains why I was having issues 🙂 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2020, 3:50am UTC](https://discuss.elastic.co/t/how-to-import-10-millions-of-documents-about-5g-into-app-search/211716/15 "2020-02-05T03:50:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.