# App search crashing after few million records **URL:** \ **Category:** Elastic Search\ **Tags:** elastic-app-search\ **Created:** [June 24, 2020, 1:02pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470 "2020-06-24T13:02:01Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Srini12](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@Srini12](https://discuss.elastic.co/u/Srini12)\ **Post date:** [June 24, 2020, 1:02pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470/1 "2020-06-24T13:02:02Z")
We have set up app-search self managed instance on a linux machine. Which is fine however it is crashing after few millions of records, below are the issues 1. We were able to write upto 14 mill records, after that the app-search is crashing with too many open connections errors. I have rebooted the server but no luck. 2. We are writing data into app search with .Net core application which is running on the same machine, the job does two parts fetches data from a document based db -- upto to here it is very quick. the second part is writing data to app-search.. which is taking too long.. more frustrated thing is it is processing only 100,000 records per day. 3. logstasher.log is growing very rapidly.. is there any way we can restrict this.? 4. Is there any limit in the app-search ? Any suggestions please. Thank you in advance. ---
**Author:** ![JasonStoltz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasonstoltz/32/49893_2.png) [@JasonStoltz](https://discuss.elastic.co/u/JasonStoltz)\ **Post date:** [July 1, 2020, 12:32pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470/2 "2020-07-01T12:32:34Z")
@Srini12 > We were able to write upto 14 mill records, after that the app-search is crashing with too many open connections errors. I have rebooted the server but no luck. Are you able to provide us with a more specific error message? > which is taking too long.. more frustrated thing is it is processing only 100,000 records per day. How are you writing the records into App Search? > logstasher.log is growing very rapidly.. is there any way we can restrict this.? EDITED: Removed my advice in favor of @orhantoy's response below > Is there any limit in the app-search ? There are no document limits in App Search ---
**Author:** ![orhantoy](https://avatars.discourse-cdn.com/v4/letter/o/c5a1d2/32.png) [@orhantoy](https://discuss.elastic.co/u/orhantoy)\ **Post date:** [July 1, 2020, 12:51pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470/3 "2020-07-01T12:51:31Z")
Hi @Srini12 👋 Reg 1. The documents are indexed into Elasticsearch - have you looked into the health of your Elasticsearch cluster? Reg 3. What version of App Search are you on, as I recall this being fixed not too long ago? ---
**Author:** ![oleksiy-elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiy-elastic/32/49074_2.png) [@oleksiy-elastic](https://discuss.elastic.co/u/oleksiy-elastic)\ **Post date:** [July 1, 2020, 3:12pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470/4 "2020-07-01T15:12:37Z")
Hello Srinivas, Sorry you're experiencing issues with indexing large amounts of data into App Search. I hope we can help you resolve those issues ASAP. As Jason and Orhan has already pointed out, we may need a bit more information to troubleshoot your specific situation. > We were able to write upto 14 mill records, after that the app-search is crashing with too many open connections errors. I have rebooted the server but no luck. This is concerning since we've never seen anything like that before and it'd be extremely helpful to get more information on the following: 1. What specific errors are you seeing and where? 2. Could you run the following command on the server running App Search when the issue is happening? > it is processing only 100,000 records per day Ingest rates are extremely dependent on the code performing the ingestion. If you need to ingest a lot of data relatively quickly, you need to ensure you're using batch indexing requests (up to 100 docs in one batch), using multiple parallel processing requests (processes, threads, etc depending on the code that does the ingestion) and are monitoring the health of your Elasticserarch cluster to ensure it is able to keep up with the data you're pushing into it. I'm fairly certain App Search could handle a lot more in a day than the numbers you're seeing, so I'd recommend not settling for that number and looking into ways to dramatically increase it. > Is there any limit in the app-search There are no limits on the number of records indexed into the system or on the ingestion rate. It all depends on available resources and proper sizing of components (mainly Elasticsearch). Thank you for using our product! -- Oleksiy Kovyrin App Search Tech Lead ---
**Author:** ![oleksiy-elastic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiy-elastic/32/49074_2.png) [@oleksiy-elastic](https://discuss.elastic.co/u/oleksiy-elastic)\ **Post date:** [July 1, 2020, 3:15pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470/5 "2020-07-01T15:15:02Z")
Oh, and one more thing: if you are able to upgrade to the latest version (Enterprise Search 7.8.0 at the moment), please do – the product is relatively young and we've made tremendous improvements over the past few minor versions (like the logstasher log, that does not exist anymore and all other logs have a proper automatic log rotation configured). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 29, 2020, 3:15pm UTC](https://discuss.elastic.co/t/app-search-crashing-after-few-million-records/238470/6 "2020-07-29T15:15:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.