# Very slow bulk indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [June 11, 2013, 12:44pm UTC](https://discuss.elastic.co/t/very-slow-bulk-indexing/12361 "2013-06-11T12:44:12Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![rohit\_reddy](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@rohit\_reddy](https://discuss.elastic.co/u/rohit_reddy)\ **Post date:** [June 11, 2013, 12:44pm UTC](https://discuss.elastic.co/t/very-slow-bulk-indexing/12361/1 "2013-06-11T12:44:12Z")
Hi, We have a elasticsearch cluster deployed on AWS. We use the following configurations: 2 m1.large instances Shards = 3 replication = 1 JVM Memory - 4.5 GB in /etc/security/limits.conf, we have set the limits to : nofile 32000. We have 2 different indexes, each of 15GB. Our use case requires us to index the entire data at one time. After the initial indexing, we don't get too many updates to the index (about 200-300 updates per day). Most of the search requests to the cluster. The first index got indexed in 2 hrs and the memory was fine during the indexing. We are facing a problem while indexing the 2nd index: There are too many old gc's happening during the indexing, We are getting a old gen every 1 min. After the old gen, the memory goes to below 1 GB, but fills up very fast. Any help is appreciated. We got the following data using JMAP: num #instances#bytes Class description * * * 1: 5940691 332678696 org.apache.lucene.index.SegmentNorms 2: 8941204 286118528 java.util.HashMap$Entry 3: 2629325 84138400 org.apache.lucene.index.FieldInfo 4: 7462 70466928 java.util.HashMap$Entry[] 5: 253939 37475472 char[] 6: 32018 32550872 byte[] 7: 25109 30609304 int[] 8: 688280 17893328 org.elasticsearch.index.mapper.FieldMapper[] 9: 688329 16519896 java.util.Arrays$ArrayList 10: 688280 16518720 org.elasticsearch.index.mapper.FieldMappers 11: 7972 15217024 java.lang.Object[] 12: 92755 13697816 \* ConstMethodKlass -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![rohit\_reddy](https://avatars.discourse-cdn.com/v4/letter/r/ed655f/32.png) [@rohit\_reddy](https://discuss.elastic.co/u/rohit_reddy)\ **Post date:** [June 13, 2013, 8:53am UTC](https://discuss.elastic.co/t/very-slow-bulk-indexing/12361/2 "2013-06-13T08:53:33Z")
Problem was bad mapping file. Fixing the mapping file, solved the issue Thanks Rohit On Tuesday, June 11, 2013 6:14:12 PM UTC+5:30, rohit reddy wrote: > Hi, > > We have a elasticsearch cluster deployed on AWS. We use the following > configurations: > > 2 m1.large instances > Shards = 3 > replication = 1 > JVM Memory - 4.5 GB > > in /etc/security/limits.conf, we have set the limits to : nofile 32000. > > We have 2 different indexes, each of 15GB. Our use case requires us to > index the entire data at one time. > After the initial indexing, we don't get too many updates to the index > (about 200-300 updates per day). Most of the search requests to the cluster. > > The first index got indexed in 2 hrs and the memory was fine during the > indexing. > > We are facing a problem while indexing the 2nd index: > There are too many old gc's happening during the indexing, We are getting > a old gen every 1 min. After the old gen, the memory goes to below 1 GB, > but fills up very fast. > > Any help is appreciated. > > We got the following data using JMAP: > > num #instances#bytes Class description > > * * * > > 1: 5940691 332678696 > org.apache.lucene.index.SegmentNorms > > 2: 8941204 286118528 java.util.HashMap$Entry > > 3: 2629325 84138400 org.apache.lucene.index.FieldInfo > > 4: 7462 70466928 java.util.HashMap$Entry > > 5: 253939 37475472 char > > 6: 32018 32550872 byte > > 7: 25109 30609304 int > > 8: 688280 17893328 > org.elasticsearch.index.mapper.FieldMapper > > 9: 688329 16519896 java.util.Arrays$ArrayList > > 10: 688280 16518720 > org.elasticsearch.index.mapper.FieldMappers > > 11: 7972 15217024 java.lang.Object > > 12: 92755 13697816 \* ConstMethodKlass -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:31am UTC](https://discuss.elastic.co/t/very-slow-bulk-indexing/12361/3 "2017-07-06T02:31:27Z")