# Indexing large number of files each with a huge size
**URL:** https://discuss.elastic.co/t/indexing-large-number-of-files-each-with-a-huge-size/19439
**Category:** Elasticsearch
**Created:** [August 25, 2014, 5:00pm UTC](https://discuss.elastic.co/t/indexing-large-number-of-files-each-with-a-huge-size/19439 "2014-08-25T17:00:39Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@ElasticSearch\_Users\_](https://discuss.elastic.co/u/ElasticSearch_Users_)
#### Post date: [August 25, 2014, 5:00pm UTC](https://discuss.elastic.co/t/indexing-large-number-of-files-each-with-a-huge-size/19439/1 "2014-08-25T17:00:39Z")
Hi,
I am trying to index documents, each file approx ~10-20 MB. I start seeing
memory issues if I try to index them all in a multi-threaded environment
from a single TransportClient on one machine to a single node cluster with
32GB ES server. It seems like the memory is an issue on the client as well
as server side, and I probably understand and expect that :).
I have tried tuning the heap sizes and batch sizes in Bulk APIs. However,
am I trying to push the limits too much? One thought is to probably stream
the data so that I do not hold it all in memory. Is it possible? Is this a
general problem or just that my usage is wrong?
Thanks,
Sandeep
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [August 25, 2014, 5:10pm UTC](https://discuss.elastic.co/t/indexing-large-number-of-files-each-with-a-huge-size/19439/2 "2014-08-25T17:10:52Z")
Can you show the program how you index?
Before tuning heap sizes or batch sizes, it is good to check if the program
works correct.
Jörg
On Mon, Aug 25, 2014 at 7:00 PM, 'Sandeep Ramesh Khanzode' via
elasticsearch [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) wrote:
> Hi,
>
> I am trying to index documents, each file approx ~10-20 MB. I start seeing
> memory issues if I try to index them all in a multi-threaded environment
> from a single TransportClient on one machine to a single node cluster with
> 32GB ES server. It seems like the memory is an issue on the client as well
> as server side, and I probably understand and expect that :).
>
> I have tried tuning the heap sizes and batch sizes in Bulk APIs. However,
> am I trying to push the limits too much? One thought is to probably stream
> the data so that I do not hold it all in memory. Is it possible? Is this a
> general problem or just that my usage is wrong?
>
> Thanks,
> Sandeep
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG7oByjnRhFoHboLJRRzhdBbsr%2BXC8NO0JU9KP0VEU4HQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG7oByjnRhFoHboLJRRzhdBbsr%2BXC8NO0JU9KP0VEU4HQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Sandeep\_Ramesh\_Khanz](https://discuss.elastic.co/u/Sandeep_Ramesh_Khanz)
#### Post date: [August 26, 2014, 7:20am UTC](https://discuss.elastic.co/t/indexing-large-number-of-files-each-with-a-huge-size/19439/3 "2014-08-26T07:20:38Z")
Hi Jorg,
This is mostly standard code that I am referring. This is called from
multiple threads for a different set of files on disk.
Please provide your suggestions. Thanks,
============================================================================================
BulkRequestBuilder bulkRequest = client.prepareBulk();
bulkRequest.setRefresh(false);
```
for every input file in the input list, do ...
Map jsonDocument = new HashMap();
```
jsonDocument.put("fileContent", );
jsonDocument.put("fileProperty1",
```
\);
jsonDocument.put("fileProperty1", \);
jsonDocument.put("fileProperty1", \);
jsonDocument.put("filePath", new
BytesRef(filePath.toString()));
```
bulkRequest.add(client.prepareIndex(indexName,
```
typeName).setSource(jsonDocument));
}
```
BulkResponse bulkResponse = bulkRequest.execute().actionGet();
```
============================================================================================
Thanks,
Sandeep
On Mon, Aug 25, 2014 at 10:40 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<
[joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote:
> Can you show the program how you index?
>
> Before tuning heap sizes or batch sizes, it is good to check if the
> program works correct.
>
> Jörg
>
> On Mon, Aug 25, 2014 at 7:00 PM, 'Sandeep Ramesh Khanzode' via
> elasticsearch [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) wrote:
>
> > Hi,
> >
> > I am trying to index documents, each file approx ~10-20 MB. I start
> > seeing memory issues if I try to index them all in a multi-threaded
> > environment from a single TransportClient on one machine to a single node
> > cluster with 32GB ES server. It seems like the memory is an issue on the
> > client as well as server side, and I probably understand and expect that
> > :).
> >
> > I have tried tuning the heap sizes and batch sizes in Bulk APIs. However,
> > am I trying to push the limits too much? One thought is to probably stream
> > the data so that I do not hold it all in memory. Is it possible? Is this a
> > general problem or just that my usage is wrong?
> >
> > Thanks,
> > Sandeep
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> >
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d2612109-b31c-4127-857b-f8aa27fb0aeb%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> [https://groups.google.com/d/topic/elasticsearch/QQDTzCAMQyU/unsubscribe](https://groups.google.com/d/topic/elasticsearch/QQDTzCAMQyU/unsubscribe).
> To unsubscribe from this group and all its topics, send an email to
> [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG7oByjnRhFoHboLJRRzhdBbsr%2BXC8NO0JU9KP0VEU4HQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG7oByjnRhFoHboLJRRzhdBbsr%2BXC8NO0JU9KP0VEU4HQ%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG7oByjnRhFoHboLJRRzhdBbsr%2BXC8NO0JU9KP0VEU4HQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoG7oByjnRhFoHboLJRRzhdBbsr%2BXC8NO0JU9KP0VEU4HQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKnM90Z91arY3mtT3QGJJow49rRdR9zawuEmTABdVC5m\_v%2B%2BuA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKnM90Z91arY3mtT3QGJJow49rRdR9zawuEmTABdVC5m_v%2B%2BuA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:06am UTC](https://discuss.elastic.co/t/indexing-large-number-of-files-each-with-a-huge-size/19439/4 "2017-07-06T01:06:34Z")