# SystemMemoryOutOfException While Indexing Documents as attachments **URL:** \ **Category:** Elasticsearch\ **Created:** [May 27, 2016, 6:58am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119 "2016-05-27T06:58:49Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 6:58am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/1 "2016-05-27T06:58:49Z")
Hi All, I'm facing a problem while trying to index documents as attachments. (NEST and VS) I am able to index documents of number 20-25 without any problem. But if I have more documents say 50 odd then I'm facing a problem with indexing. I'm using `Index.Many` to index the documents. My Understanding: According to what I understood, those 20 odd documents are of 5mb in total where as the remaining documents 30 odd or so, are of 80 mb in which one file is 50mb. I think its throwing the error when it is trying to index this big document. But previously when i worked on the something similar, I was able to index this document too without any problem. More description about the issue > [@SystemMemoryOutofException thrown while indexing files as an attachment](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916): > > Hi all, I'm getting an error while using IndexMany for indexing documents as attachments. Yesterday, When I published the application on server it worked fine. But when I try to run it from visual studio it is throwing this error. Can someone help me in solving this. string path = "D:\ElasticShare\"; List\ filesList = new List\(); try { string[] filePath = Directory.GetFiles(path, "\*", SearchOption.AllDirectories); … ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [May 27, 2016, 7:59am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/2 "2016-05-27T07:59:37Z")
That's a pretty big document, but shouldn't really be an issue... I can think of a couple of things you can try: - Can you run the application outside of Visual Studio and see what it does - Can you run the application without indexing that single large document and see what it does - Can you look at your mem usage in Task Manager while running in Visual Studio and see how much memory it actually uses If you're using Visual Studio 2015 you can also crack out the [diagnostic tools window](https://blogs.msdn.microsoft.com/visualstudioalm/2015/01/16/diagnostic-tools-debugger-window-in-visual-studio-2015/) and see what it's up to. EDIT: Looking at the code you put in the other thread it looks like the issue specifically is while serialising the content encoding... Is there any particular reason you're using `Base64` and not `UTF8` for the file data? But I'm not totally sure how much difference that makes... ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 8:23am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/3 "2016-05-27T08:23:43Z")
Thanks for the reply Ashley. Here are the observations you have asked for. > [@KodrAus](#): > > Can you run the application outside of Visual Studio and see what it does I'm assuming that you are asking me to run the .exe of the application and If it is that I did that _with the document_. It didn't throw any error(as you can see, I wrote the index call in try-catch block. It didnt come to catch block.) I had a success message after indexing. So when I ran the .exe, no error was found and the next success message was printed, but no files are indexed when i checked in the Head Plugin. > [@KodrAus](#): > > Can you run the application without indexing that single large document and see what it does _without the document and using the .exe of the application_ I could index all the document without any problem and even this is reflected in HEAD plugin. > [@KodrAus](#): > > Can you look at your mem usage in Task Manager while running in Visual Studio and see how much memory it actually uses _without that doc_ I started with an available memory of ~958mb and when I run the application, it went to the lowest of ~330mb and once I close the application it took some time to close and then it released the memory back. After coming back to the normal position the available memory is ~800mb _with that doc_ I started with an available memory of ~1.3Gband when I run the application, it went to the lowest of ~340mb , threw an error and once I close the application it closed and then it released the memory back. After coming back to the normal position the available memory is ~1.2Gb. But the cpu utilization was 100% at few times. Please let me know if any other info is required. TIA ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [May 27, 2016, 8:57am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/4 "2016-05-27T08:57:57Z")
> [@ASN](#): > > no files are indexed when i checked in the Head Plugin. That's weird... are you doing anything with the `response` object you get from `IndexMany`? Is it returning any errors to say the request didn't succeed? And I have to say... Nearly 1Gb of RAM needed to index 30Mb of data is a pretty sad ratio 😄 (not your fault, that's just .NET for you) ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 9:03am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/5 "2016-05-27T09:03:11Z")
No. I just created the variable thats it. something like ` var response = client.IndexMany(list,"indexname")`. It is throwing an error here everytime. Sometimes it work perfectly, sometimes it wont. Confused . Dont know if it is my code problem or elasticsearch problem or as u said .NET problem(😋). Adding to this problem, I have another problem. > [@FileSystemwatcher with Documents Indexing as Attachments (c#,Nest)](https://discuss.elastic.co/t/filesystemwatcher-with-documents-indexing-as-attachments-c-nest/51105): > > Hi All, I'm trying to reproduce a console application which watches a folder and any new additions of documents to the folder are to be Indexed to ES . It is working fine If I move/add 3-4 documents at a time and able to index. But if I move around 30 documents at a time, It is not indexing all the documents, instead indexing only one. But If I run the code with break points , then even 30 documents are also getting indexed. Can some one help me in solving this. static void OnCreated(object… So this week, I'm completely stuck with these issues. 😔 ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [May 27, 2016, 9:20am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/6 "2016-05-27T09:20:16Z")
Nest doesn't always throw an exception when there's a problem with the request. So in the case where you get a success message but nothing is indexed I would check that `response` to make sure it actually succeeded. There's some properties on there (I can't remember the name off the top of my head) that contains the actual response from Elasticsearch. You can write that to console to see what you're getting back when not running from Visual Studio. How much memory does your app use when you run the exe without Visual Studio debugging? Should be much less... ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 9:23am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/7 "2016-05-27T09:23:17Z")
Response has some properties like ![](https://us1.discourse-cdn.com/elastic/original/2X/f/f55697675de150cd40ef6d1c53160005a8801e0e.png) ![](https://us1.discourse-cdn.com/elastic/original/2X/a/a4ab7fff830977de4ab3cf7dd95599c82316a0eb.png) I printed out calldetails and original exception details. Below is the screenshot of that. ![](https://us1.discourse-cdn.com/elastic/original/2X/9/9130ac4fccf2cd4dc63947ee085e7157cbfbf9c0.png) ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [May 27, 2016, 9:31am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/8 "2016-05-27T09:31:38Z")
Does that error message happen reliably every time? Maybe the socket is being disposed because of low memory... ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 9:35am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/9 "2016-05-27T09:35:40Z")
With the Items in the response. This time it didnt print the success message. Dont know why. ![](https://us1.discourse-cdn.com/elastic/original/2X/9/9e09757dee3b4737dc14c6bf114571873f459465.png) ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [May 27, 2016, 9:38am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/10 "2016-05-27T09:38:28Z")
I'm thinking buffering all your files into memory and throwing them Elasticsearch at once isn't going so well. Before doing anything else, do you want to adjust your program a bit to only read 1 file at a time? You could do this by grabbing the file metadata like you do, but instead of reading the contents of all files into the list, read 1 file, upload to Elasticsearch, dispose of the filestream, read the next file and so on. Sorry I'd go and quickly tweak your sample code but I'm on my phone so can't edit code well 🙂 ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [May 27, 2016, 9:58am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/12 "2016-05-27T09:58:44Z")
Yeh if it's not private then feel free to make a github gist or something with the sample and send a link. I should get some time over the weekend to take a look 😄 ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 2:30am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/14 "2016-05-30T02:30:37Z")
Hi Ashley, While I'm indexing documents, I tried to print the response. ``` var response = client.IndexMany(list, "watcherfilesreader"); foreach (var item in response.Items) { Console.WriteLine(item.ToString()); } ``` ![](https://us1.discourse-cdn.com/elastic/original/2X/6/6118adc71555eb9e34cdb1438e182abab754bbea.png) It says 1 error, but displays nothing. Is it correctly indexed or indexed with error? TIA ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [June 8, 2016, 1:27am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/15 "2016-06-08T01:27:15Z")
I'm still having the memoryoutofexception issues...I'm not sure if it is happening on VS side or elastic search side. ![](https://us1.discourse-cdn.com/elastic/original/2X/a/aee1dd3093a8a6c91e14d26b7439cb11dc114464.png) ---
**Author:** ![KodrAus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kodraus/32/22798_2.png) [@KodrAus](https://discuss.elastic.co/u/KodrAus)\ **Post date:** [June 8, 2016, 6:08am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/16 "2016-06-08T06:08:01Z")
Hmmm, that would be on the VS side, it looks like memory is a bit of a sticking point on your setup, especially with that large document. Both .NET and Elasticsearch are fairly free-spirited with memory and will take as much as they can get. Memory debugging can be a bit challenging, especially when the issue is that you're running out of it, but Visual Studio's [memory diagnostic tools](https://blogs.msdn.microsoft.com/visualstudioalm/2014/04/02/diagnosing-memory-issues-with-the-new-memory-usage-tool-in-visual-studio/) might be able to help you out. Otherwise something more heavy-weight like [ANTS Memory Profiler](http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/) (has a free trial) could be worth looking into. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:45pm UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-while-indexing-documents-as-attachments/51119/17 "2017-07-05T22:45:26Z")