# Loading data on ec2 micro instances **URL:** \ **Category:** Elasticsearch\ **Created:** [April 16, 2012, 2:23am UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339 "2012-04-16T02:23:40Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Ranjan\_Bagchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjan_bagchi/32/2918_2.png) [@Ranjan\_Bagchi](https://discuss.elastic.co/u/Ranjan_Bagchi)\ **Post date:** [April 16, 2012, 2:23am UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/1 "2012-04-16T02:23:40Z")
Hi, I've a file of several million documents that I'd like to load into an elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. What's the best way to load the index up? I'm using another machine and the java client, but it's really taking a long time and I'm finding the instances in the cluster keep falling over. Any help? Thanks, Ranjan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 17, 2012, 1:06pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/2 "2012-04-17T13:06:37Z")
Don't use micro instances On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi [ranjan.bagchi@gmail.com](mailto:ranjan.bagchi@gmail.com)wrote: > Hi, > > I've a file of several million documents that I'd like to load into an > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > What's the best way to load the index up? I'm using another machine and > the java client, but it's really taking a long time and I'm finding the > instances in the cluster keep falling over. > > Any help? > > Thanks, > > Ranjan ---
**Author:** ![Ranjan\_Bagchi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjan_bagchi/32/2918_2.png) [@Ranjan\_Bagchi](https://discuss.elastic.co/u/Ranjan_Bagchi)\ **Post date:** [April 18, 2012, 2:33pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/3 "2012-04-18T14:33:26Z")
So I ended up solving the problem by doing the following to each node: Shrinking the max heap to 450M Adding swap on the EBS node as follows: \* /bin/dd if=/dev/zero of=/ebs/swap.1 bs=1M count=1024 \* mkswap /ebs/swap.1 \* swapon /ebs/swap.1 I was able to add 12M small documents in several hours smoothly after that. Ranjan On Tuesday, April 17, 2012 6:06:37 AM UTC-7, kimchy wrote: > Don't use micro instances > > On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi [ranjan.bagchi@gmail.com](mailto:ranjan.bagchi@gmail.com)wrote: > > > Hi, > > > > I've a file of several million documents that I'd like to load into an > > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > > > What's the best way to load the index up? I'm using another machine and > > the java client, but it's really taking a long time and I'm finding the > > instances in the cluster keep falling over. > > > > Any help? > > > > Thanks, > > > > Ranjan ---
**Author:** ![Clay\_Graham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clay_graham/32/2900_2.png) [@Clay\_Graham](https://discuss.elastic.co/u/Clay_Graham)\ **Post date:** [April 18, 2012, 4:29pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/4 "2012-04-18T16:29:11Z")
It seems counter intuitive to me that Elasticsearch is not recommended for a micro profile. Seems more a question of tuning and shard planning to me. On Tuesday, April 17, 2012 6:06:37 AM UTC-7, kimchy wrote: > Don't use micro instances > > On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi [ranjan.bagchi@gmail.com](mailto:ranjan.bagchi@gmail.com)wrote: > > > Hi, > > > > I've a file of several million documents that I'd like to load into an > > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > > > What's the best way to load the index up? I'm using another machine and > > the java client, but it's really taking a long time and I'm finding the > > instances in the cluster keep falling over. > > > > Any help? > > > > Thanks, > > > > Ranjan ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [April 18, 2012, 5:28pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/5 "2012-04-18T17:28:17Z")
What is it counter intuitive? Amazon states that micro instances are "well suited for lower throughput applications and web sites that require additional compute cycles periodically". Amazon documentation shows the type of workloads that would not be suitable for micro instances, and most ES uses (if not all) don't have behavior that would match the behavior patter described by Amazon. > **[Burstable performance instances - Amazon Elastic Compute Cloud](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/burstable-performance-instances.html)** > > Use these instances for baseline performance and the ability to burst to higher performance as required by your workload. It may be possible to run ES on micro instances with proper tuning, but it's likely not a good idea to use them in production. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Wed, Apr 18, 2012 at 12:29 PM, Clay Graham [clay@welocally.com](mailto:clay@welocally.com) wrote: > It seems counter intuitive to me that Elasticsearch is not recommended > for a micro profile. Seems more a question of tuning and shard planning to > me. > > On Tuesday, April 17, 2012 6:06:37 AM UTC-7, kimchy wrote: > > > Don't use micro instances > > > > On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi [ranjan.bagchi@gmail.com](mailto:ranjan.bagchi@gmail.com)wrote: > > > > > Hi, > > > > > > I've a file of several million documents that I'd like to load into an > > > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > > > > > What's the best way to load the index up? I'm using another machine and > > > the java client, but it's really taking a long time and I'm finding the > > > instances in the cluster keep falling over. > > > > > > Any help? > > > > > > Thanks, > > > > > > Ranjan ---
**Author:** ![Clay\_Graham](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/clay_graham/32/2900_2.png) [@Clay\_Graham](https://discuss.elastic.co/u/Clay_Graham)\ **Post date:** [April 25, 2012, 8:27pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/6 "2012-04-25T20:27:03Z")
Well saying "dont use micro instances" is way different than "you may have to tune them". please explain "likely not a good idea". Based on what values? This is not me arguing, this is honest curiosity. What makes me think its counter intuitive is that the whole point of horizontal scaling is that its well... horizontal. We are a _lean_ startup, we have millions of docs not millions of users, and the cost profile for running at the next instance tier up is way more expensive. Right now we are running a four node cluster so it costs about $40/month to run our cluster as micros and it performs pretty well as long as we add swap. I am also pretty sure it runs as good if not better than the same cluster with small instances and two nodes (more expensive). we would love to be able to afford a better cluster but unless there is a need we would prefer to run lean. We also want to do lean geospatial search projects for non-profits like Code for Oakland that may just not be willing to do projects that require major overhead. So if this is a nonstarter we would love to know. Respectfully, Clay On Wednesday, April 18, 2012 10:28:17 AM UTC-7, Berkay Mollamustafaoglu wrote: > What is it counter intuitive? Amazon states that micro instances are "well > suited for lower throughput applications and web sites that require > additional compute cycles periodically". Amazon documentation shows the > type of workloads that would not be suitable for micro instances, and most > ES uses (if not all) don't have behavior that would match the behavior > patter described by Amazon. > > [Burstable performance instances - Amazon Elastic Compute Cloud](http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html) > > It may be possible to run ES on micro instances with proper tuning, but > it's likely not a good idea to use them in production. > > Regards, > Berkay Mollamustafaoglu > mberkay on yahoo, google and skype > > On Wed, Apr 18, 2012 at 12:29 PM, Clay Graham [clay@welocally.com](mailto:clay@welocally.com) wrote: > > > It seems counter intuitive to me that Elasticsearch is not recommended > > for a micro profile. Seems more a question of tuning and shard planning to > > me. > > > > On Tuesday, April 17, 2012 6:06:37 AM UTC-7, kimchy wrote: > > > > > Don't use micro instances > > > > > > On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi wrote: > > > > > > > Hi, > > > > > > > > I've a file of several million documents that I'd like to load into an > > > > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > > > > > > > What's the best way to load the index up? I'm using another machine > > > > and the java client, but it's really taking a long time and I'm finding the > > > > instances in the cluster keep falling over. > > > > > > > > Any help? > > > > > > > > Thanks, > > > > > > > > Ranjan ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [April 25, 2012, 9:48pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/7 "2012-04-25T21:48:36Z")
I have run test instances of ES on microinstances successfully. Had some spares around (used for low intensity processes) so I started 1-3 node clusters. Your issue is with startup indexing? If you are using bulk indexing, I would keep the number very low (depends on your document size). That said, I would not recommend using microinstances. For searching, I would avoid any type of faceted search. Disable as many indexed/term-vectors/norms as possible. -- Ivan On Wed, Apr 25, 2012 at 1:27 PM, Clay Graham [clay@welocally.com](mailto:clay@welocally.com) wrote: > Well saying "dont use micro instances" is way different than "you may have > to tune them". please explain "likely not a good idea". Based on what > values? This is not me arguing, this is honest curiosity. What makes me > think its counter intuitive is that the whole point of horizontal scaling is > that its well... horizontal. > > We are a _lean_ startup, we have millions of docs not millions of users, and > the cost profile for running at the next instance tier up is way more > expensive. Right now we are running a four node cluster so it costs about > $40/month to run our cluster as micros and it performs pretty well as long > as we add swap. I am also pretty sure it runs as good if not better than the > same cluster with small instances and two nodes (more expensive). we would > love to be able to afford a better cluster but unless there is a need we > would prefer to run lean. We also want to do lean geospatial search projects > for non-profits like Code for Oakland that may just not be willing to do > projects that require major overhead. > > So if this is a nonstarter we would love to know. > > Respectfully, > > Clay > > On Wednesday, April 18, 2012 10:28:17 AM UTC-7, Berkay Mollamustafaoglu > wrote: > > > What is it counter intuitive? Amazon states that micro instances are "well > > suited for lower throughput applications and web sites that require > > additional compute cycles periodically". Amazon documentation shows the type > > of workloads that would not be suitable for micro instances, and most ES > > uses (if not all) don't have behavior that would match the behavior patter > > described by Amazon. > > > > [Burstable performance instances - Amazon Elastic Compute Cloud](http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html) > > > > It may be possible to run ES on micro instances with proper tuning, but > > it's likely not a good idea to use them in production. > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Wed, Apr 18, 2012 at 12:29 PM, Clay Graham [clay@welocally.com](mailto:clay@welocally.com) wrote: > > > > > It seems counter intuitive to me that Elasticsearch is not recommended > > > for a micro profile. Seems more a question of tuning and shard planning to > > > me. > > > > > > On Tuesday, April 17, 2012 6:06:37 AM UTC-7, kimchy wrote: > > > > > > > Don't use micro instances > > > > > > > > On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi wrote: > > > > > > > > > Hi, > > > > > > > > > > I've a file of several million documents that I'd like to load into an > > > > > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > > > > > > > > > What's the best way to load the index up? I'm using another machine > > > > > and the java client, but it's really taking a long time and I'm finding the > > > > > instances in the cluster keep falling over. > > > > > > > > > > Any help? > > > > > > > > > > Thanks, > > > > > > > > > > Ranjan ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [April 26, 2012, 3:11pm UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/8 "2012-04-26T15:11:19Z")
The shortcomings of the micro instances are explained in the Amazon doc I've referenced in the previous email. These are essentially highly oversubscribed VMs. It's not only a matter of configuration parameters on your part; it's also highly impacted by the activities of the other instances on the same server. This leads me to believe that performance would be very unpredictable and user experience can suffer, especially when usage is not uniform. If it works for your use cases, great. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Wed, Apr 25, 2012 at 4:27 PM, Clay Graham [clay@welocally.com](mailto:clay@welocally.com) wrote: > Well saying "dont use micro instances" is way different than "you may have > to tune them". please explain "likely not a good idea". Based on what > values? This is not me arguing, this is honest curiosity. What makes me > think its counter intuitive is that the whole point of horizontal scaling > is that its well... horizontal. > > We are a _lean_ startup, we have millions of docs not millions of users, > and the cost profile for running at the next instance tier up is way more > expensive. Right now we are running a four node cluster so it costs about > $40/month to run our cluster as micros and it performs pretty well as long > as we add swap. I am also pretty sure it runs as good if not better than > the same cluster with small instances and two nodes (more expensive). we > would love to be able to afford a better cluster but unless there is a need > we would prefer to run lean. We also want to do lean geospatial search > projects for non-profits like Code for Oakland that may just not be willing > to do projects that require major overhead. > > So if this is a nonstarter we would love to know. > > Respectfully, > > Clay > > On Wednesday, April 18, 2012 10:28:17 AM UTC-7, Berkay Mollamustafaoglu > wrote: > > > What is it counter intuitive? Amazon states that micro instances are "well > > suited for lower throughput applications and web sites that require > > additional compute cycles periodically". Amazon documentation shows the > > type of workloads that would not be suitable for micro instances, and most > > ES uses (if not all) don't have behavior that would match the behavior > > patter described by Amazon. > > [http://docs.amazonwebservices](http://docs.amazonwebservices). **com/AWSEC2/latest/UserGuide/** > > concepts\_micro\_instances.html[http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts\_micro\_instances.html](http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html) > > > > It may be possible to run ES on micro instances with proper tuning, but > > it's likely not a good idea to use them in production. > > > > Regards, > > Berkay Mollamustafaoglu > > mberkay on yahoo, google and skype > > > > On Wed, Apr 18, 2012 at 12:29 PM, Clay Graham [clay@welocally.com](mailto:clay@welocally.com) wrote: > > > > > It seems counter intuitive to me that Elasticsearch is not recommended > > > for a micro profile. Seems more a question of tuning and shard planning to > > > me. > > > > > > On Tuesday, April 17, 2012 6:06:37 AM UTC-7, kimchy wrote: > > > > > > > Don't use micro instances > > > > > > > > On Mon, Apr 16, 2012 at 5:23 AM, Ranjan Bagchi wrote: > > > > > > > > Hi, > > > > > > > > > I've a file of several million documents that I'd like to load into an > > > > > elasticsearch cluster, hosted on 4 ec2 micro instances using the S3 gateway. > > > > > > > > > > What's the best way to load the index up? I'm using another machine > > > > > and the java client, but it's really taking a long time and I'm finding the > > > > > instances in the cluster keep falling over. > > > > > > > > > > Any help? > > > > > > > > > > Thanks, > > > > > > > > > > Ranjan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:31am UTC](https://discuss.elastic.co/t/loading-data-on-ec2-micro-instances/7339/9 "2017-07-06T03:31:02Z")