# SystemMemoryOutofException thrown while indexing files as an attachment **URL:** \ **Category:** Elasticsearch\ **Created:** [May 25, 2016, 7:48am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916 "2016-05-25T07:48:15Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 7:15am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/22 "2016-05-27T07:15:35Z")
okok. I tried my best to keep it simple and understandable. And one more doubt David. Why is the size of files increased when indexed ? The total size of the folder from which documents are indexed is 30mb. But the head plugin is showing 127mb for the same number of files(which are indexed from the same folder) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 27, 2016, 7:52am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/23 "2016-05-27T07:52:54Z")
It depends. But basically here are my thoughts. When you index a doc (using defaults), elasticsearch is: - indexing all the fields one by one - storing in `_source` the JSON - creating doc values data structure (column oriented data structure for aggregations) - creating on the fly a flat version of your JSON doc in `_all` field and index it Then add to this analyzers. If you are using multi fields, it can become worse. On advice: control your mapping and adjust it wisely. - You may be don't need to index everything. - You may be don't want to store the binary BASE64 content in your `_source` field: that's why we created ingest-attachment plugin in 5.0 basically. - You may want to disable `_all` field... HTH ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 8:00am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/24 "2016-05-27T08:00:50Z")
But I read that if we dont store the binary BASE64 content in \_source field we cannot use the highlight option. This is my mapping style ``` { "mappings": { "document": { "properties": { "title": { "type": "string" }, "file": { "type": "attachment", "fields": { "content": { "type": "string", "store": true, "term_vector": "with_positions_offsets_payloads", "analyzer": "english" } } } } } } } ``` I havent used any of the \_all field or \_source field ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 27, 2016, 8:50am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/25 "2016-05-27T08:50:53Z")
The only way ATM to remove the BASE64 from the document is to [source exclude](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-source-field.html#include-exclude). Note that you are highlighting on the extracted content `file.content` which is generated at index time but you are not highlighting on `file` which has the BASE64 text. So you are also storing `file.content` which requires more space obviously even if it's compressed on disk. `_all` is enabled by default. [https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-all-field.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/mapping-all-field.html) ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 8:55am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/26 "2016-05-27T08:55:28Z")
oh ok. Will make the changed in my mapping. David, did you look at github for fscrawler? I'm still facing that issue. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 27, 2016, 9:10am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/27 "2016-05-27T09:10:55Z")
> [@ASN](#): > > David, did you look at github for fscrawler? I'm still facing that issue. Nope. Lot of stuff on my plates ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 27, 2016, 9:11am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/28 "2016-05-27T09:11:23Z")
Ohh okok.. No problem.. Thanks for the help all this week. ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 3:33am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/29 "2016-05-30T03:33:10Z")
hi David, I observed this while indexing the documents. (Dont know if I'm correct with this) When I indexed the documents using manual id, the size is around 36mb but when I remove the Id field and index(auto generate the id), then it is taking so much time to index, the size is more and also the search function is not working properly. Does it depend on how the file is indexed?) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 5:10am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/30 "2016-05-30T05:10:40Z")
Do you have an example? ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 5:39am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/31 "2016-05-30T05:39:00Z")
Hi David, Here is the code how I'm giving manual id's for the documents indexed. But when I comment out Id field in all the occurrences in the below snaps, It is supposed to be autogenerated and it is autogenerating.. ``` class Document { public int Id { get; set; } public string Title { get; set; } public string FilePath { get; set; } public Attachment File { get; set; } } ``` Mappings: ``` var createIndexResponse = client.CreateIndex(defaultIndex, c => c .Mappings(m => m .Map(mp => mp .Properties(ps => ps .Number(n => n.Name(e => e.Id)) .String(s => s.Name(e => e.Title)) ``` While Indexing: (where counter =1 on start) ``` foreach (string file in filesList) { Attachment attach = new Attachment { Name = Path.GetFileNameWithoutExtension(file), Content = Convert.ToBase64String(File.ReadAllBytes(file)), ContentType = Path.GetExtension(file) }; var doc = new Document() { Id = counter, Title = Path.GetFileNameWithoutExtension(file), FilePath = Path.GetFullPath(file), //added to get the path of the file File = attach }; list.Add(doc); counter++; } var response = client.IndexMany(list, "indexname"); ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 5:52am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/32 "2016-05-30T05:52:17Z")
I don't know what all that code is doing when calling Elasticsearch. Is this id the same as `_id` we have in Elasticsearch as a metadata field? ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 5:56am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/33 "2016-05-30T05:56:52Z")
Once i made a search , this is what I see as a result in sense plugin. I made \_id same as the document id I'm entering in the above code snippet. ![](https://us1.discourse-cdn.com/elastic/original/2X/5/56843ce21564cd3b9859ec71c3c2b9309443ee5f.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 6:14am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/34 "2016-05-30T06:14:24Z")
Could you reproduce this? Create a doc with id and the other one without id Without any attachment just one field. And report here? I'm surprised that automatic id are taking so much space/time. ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 7:12am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/35 "2016-05-30T07:12:18Z")
I tried it. As there are no attachments the docs got indexed in the same time. And the size differs only by 6kb (custom id-36kb where as autogen id 30kb for a total of 50docs with fields like title and filePath) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 8:33am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/36 "2016-05-30T08:33:03Z")
Can you reproduce it with a shell script? ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 8:38am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/37 "2016-05-30T08:38:21Z")
I dont know how to do it using shell script. In the above code snippets i just commented unnecessary fields and then indexed the documents and this is what I observed. I tried to index docs using sense plugin. with user defined id of 1 ``` PUT abc/docs/1 { "title":"trial1" } ``` Without user defined id ``` PUT abc/docs/ { "title":"trial1" } ``` Then it threw this error: `No handler found for uri [/abc/docs/] and method [PUT]` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 8:51am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/38 "2016-05-30T08:51:30Z")
For the later you have to use `POST` verb. ---
**Author:** ![ASN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asn/32/10535_2.png) [@ASN](https://discuss.elastic.co/u/ASN)\ **Post date:** [May 30, 2016, 8:57am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/39 "2016-05-30T08:57:22Z")
ok David. Ran it with POST. ![](https://us1.discourse-cdn.com/elastic/original/2X/5/57ed39c42d75306c2ca54ec2fa2d04f4e651913e.png) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 9:25am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/40 "2016-05-30T09:25:44Z")
Sorry if I was unclear but this is not what I was expecting from you. You told that you have a problem when you index data containing attachments. When you define the `_id` it takes a way lesser space on disk than when you don't define the `_id`. I asked to reproduce this with a full script. Which basically means create a script like: ```auto DELETE index PUT index { "mappings": ... // Your mapping here } PUT index/doc/1 { "attachment": "BASE64 content" } GET _cat/indices/index?v ``` Attach the output of the later command. Then run: ```auto DELETE index PUT index { "mappings": ... // Your mapping here } POST index/doc { "attachment": "BASE64 content" } GET _cat/indices/index?v ``` Attach the output of the later command. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [May 30, 2016, 10:14am UTC](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916/42 "2016-05-30T10:14:33Z")
This is why I asked for a reproduction. I do not believe `_id` can explain that. I think you are indexing documents with different content which would explain this. [Previous page](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916.md?page=1) [Next page](https://discuss.elastic.co/t/systemmemoryoutofexception-thrown-while-indexing-files-as-an-attachment/50916.md?page=3)