# Is it possible to index a file with a size of 26GB in elasticsearch?

**URL:** <https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831>\
**Category:** Elasticsearch\
**Created:** [April 27, 2018, 12:42pm UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831 "2018-04-27T12:42:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Michel\_Budimbu-Esuku](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michel_budimbu-esuku/32/30558_2.png) [@Michel\_Budimbu-Esuku](https://discuss.elastic.co/u/Michel_Budimbu-Esuku)\
**Post date:** [April 27, 2018, 12:42pm UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831/1 "2018-04-27T12:42:52Z")

</div>

Hello everybody  
I have a mbox file with a size of 26 GB (this file is called bigFile.mbox).  
I want to index bigFile.mbox in Elasticsearch. How can I do ?

Here's what I tried to put in place:

1.I started writing a python script that converted the bigFile.mbox file to bigFile.json:

The script works correctly when it comes to small files (\<= 1MB).  
But nothing works correctly when it comes to converting bigFile.mbox which has a size of 26GB 😥  
2. I created a python script (splitter.py) that splits an mbox file into several files of a given size:  
But this script takes a long time to cut a 26GB file and  
generates hugely small 1MB mbox files which makes indexing in  
Elasticsearch much longer.

Please, is it the most efficient way to index a 26GB mbox file in Elasticsearch?

I thank you in advance.

Best regards  
MBE

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 29, 2018, 1:23pm UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831/2 "2018-04-29T13:23:36Z")

</div>

You probably don't want to index one mailbox as a whole document but you probably want to index every single email which is in the mailbox.

My guess is that you want to be able a find an email and not a mailbox which contains XYZ.

So you need to find a way to read every single email and send it to elasticsearch. Then you can use may be ingest-attachment plugin to index each email.

My 2 cents.

---

<div class="post-metadata">

**Author:** ![Michel\_Budimbu-Esuku](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michel_budimbu-esuku/32/30558_2.png) [@Michel\_Budimbu-Esuku](https://discuss.elastic.co/u/Michel_Budimbu-Esuku)\
**Post date:** [April 30, 2018, 8:48am UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831/3 "2018-04-30T08:48:24Z")

</div>

Hello dadoonet,  
do you know an effective way to index each email that is in a mailbox?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 30, 2018, 9:02am UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831/4 "2018-04-30T09:02:18Z")

</div>

No I don't.  
May be Tika project has some code related to parsing mailbox content. It's in Java though.

---

<div class="post-metadata">

**Author:** ![Michel\_Budimbu-Esuku](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michel_budimbu-esuku/32/30558_2.png) [@Michel\_Budimbu-Esuku](https://discuss.elastic.co/u/Michel_Budimbu-Esuku)\
**Post date:** [April 30, 2018, 9:09am UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831/5 "2018-04-30T09:09:36Z")

</div>

thank you very much :)😀

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 28, 2018, 9:09am UTC](https://discuss.elastic.co/t/is-it-possible-to-index-a-file-with-a-size-of-26gb-in-elasticsearch/129831/6 "2018-05-28T09:09:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.