# Importing Large Amounts of Data to Production Indices **URL:** \ **Category:** Elasticsearch\ **Created:** [January 28, 2015, 12:45am UTC](https://discuss.elastic.co/t/importing-large-amounts-of-data-to-production-indices/21864 "2015-01-28T00:45:55Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![webish](https://avatars.discourse-cdn.com/v4/letter/w/919ad9/32.png) [@webish](https://discuss.elastic.co/u/webish)\ **Post date:** [January 28, 2015, 12:45am UTC](https://discuss.elastic.co/t/importing-large-amounts-of-data-to-production-indices/21864/1 "2015-01-28T00:45:55Z")
I have some production indices that needs to a large amount of data imported into them fairly frequently. Each time we import data the ES nodes become a huge bottleneck. I honestly expected a lot better performance out of them. Regardless, I would like to import data in a production ES setup with the least amount of interruption or performance issues. What are some options I can take to import large quantities of data without affecting data that is already being used by applications? I was thinking I could use a combination of aliases or temp indices to migrate the data over... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 28, 2015, 1:02am UTC](https://discuss.elastic.co/t/importing-large-amounts-of-data-to-production-indices/21864/2 "2015-01-28T01:02:46Z")
How much data are you talking? Are you using bulk API? What is your bulk sizing? You can also set an index to not refresh while you ingest it (refresh = -1), then once it's been sent to ES turn indexing back on. On 28 January 2015 at 11:45, webish [gregory@yoursports.com](mailto:gregory@yoursports.com) wrote: > I have some production indices that needs to a large amount of data > imported into them fairly frequently. Each time we import data the ES > nodes become a huge bottleneck. I honestly expected a lot better > performance out of them. Regardless, I would like to import data in a > production ES setup with the least amount of interruption or performance > issues. > > What are some options I can take to import large quantities of data > without affecting data that is already being used by applications? > > I was thinking I could use a combination of aliases or temp indices to > migrate the data over... > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8MuHkUdVoznQAiZFVx45nqhNngqGRrw-NxiSZH6opAvg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8MuHkUdVoznQAiZFVx45nqhNngqGRrw-NxiSZH6opAvg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![webish](https://avatars.discourse-cdn.com/v4/letter/w/919ad9/32.png) [@webish](https://discuss.elastic.co/u/webish)\ **Post date:** [January 29, 2015, 8:13am UTC](https://discuss.elastic.co/t/importing-large-amounts-of-data-to-production-indices/21864/3 "2015-01-29T08:13:19Z")
Hi Mark, Right now. 28 GB across two indices 5 shards 1 replica per index on 3 AWS large servers. Frequently 1-10 million records or more get imported. During this time all ES nodes hit a CPU usage of over 75%. We want to break the index down and add routing at some point. Refresh is using default (1) and based on coupling to some old imports system the bulk API is NOT used... Problem is the index get's accessed and written to constantly by users. So disabling refresh would delay their content from being indexed. I was debating using a separate index per import and grouping all the indices by an alias. Not certain how that will affect performance. On Tuesday, January 27, 2015 at 8:03:16 PM UTC-5, Mark Walkom wrote: > How much data are you talking? Are you using bulk API? What is your bulk > sizing? > > You can also set an index to not refresh while you ingest it (refresh = > -1), then once it's been sent to ES turn indexing back on. > > On 28 January 2015 at 11:45, webish \<[gre...@yoursports.com](mailto:gre...@yoursports.com) \\> > wrote: > > > I have some production indices that needs to a large amount of data > > imported into them fairly frequently. Each time we import data the ES > > nodes become a huge bottleneck. I honestly expected a lot better > > performance out of them. Regardless, I would like to import data in a > > production ES setup with the least amount of interruption or performance > > issues. > > > > What are some options I can take to import large quantities of data > > without affecting data that is already being used by applications? > > > > I was thinking I could use a combination of aliases or temp indices to > > migrate the data over... > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/40010b4f-852b-4c2a-81a0-4187f9b5990b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/40010b4f-852b-4c2a-81a0-4187f9b5990b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [January 29, 2015, 10:34am UTC](https://discuss.elastic.co/t/importing-large-amounts-of-data-to-production-indices/21864/4 "2015-01-29T10:34:48Z")
You should be using the bulk API, that's what it exists for! On 29 January 2015 at 19:13, webish [gregory@yoursports.com](mailto:gregory@yoursports.com) wrote: > Hi Mark, > > Right now. 28 GB across two indices 5 shards 1 replica per index on 3 AWS > large servers. > > Frequently 1-10 million records or more get imported. During this time > all ES nodes hit a CPU usage of over 75%. We want to break the index down > and add routing at some point. > > Refresh is using default (1) and based on coupling to some old imports > system the bulk API is NOT used... Problem is the index get's accessed and > written to constantly by users. So disabling refresh would delay their > content from being indexed. > > I was debating using a separate index per import and grouping all the > indices by an alias. Not certain how that will affect performance. > > On Tuesday, January 27, 2015 at 8:03:16 PM UTC-5, Mark Walkom wrote: > > > How much data are you talking? Are you using bulk API? What is your bulk > > sizing? > > > > You can also set an index to not refresh while you ingest it (refresh = > > -1), then once it's been sent to ES turn indexing back on. > > > > On 28 January 2015 at 11:45, webish [gre...@yoursports.com](mailto:gre...@yoursports.com) wrote: > > > > > I have some production indices that needs to a large amount of data > > > imported into them fairly frequently. Each time we import data the ES > > > nodes become a huge bottleneck. I honestly expected a lot better > > > performance out of them. Regardless, I would like to import data in a > > > production ES setup with the least amount of interruption or performance > > > issues. > > > > > > What are some options I can take to import large quantities of data > > > without affecting data that is already being used by applications? > > > > > > I was thinking I could use a combination of aliases or temp indices to > > > migrate the data over... > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/410c454e-7e8d-4f1b-b70a-68e18fa7c732%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/40010b4f-852b-4c2a-81a0-4187f9b5990b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/40010b4f-852b-4c2a-81a0-4187f9b5990b%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/40010b4f-852b-4c2a-81a0-4187f9b5990b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/40010b4f-852b-4c2a-81a0-4187f9b5990b%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8LJUcKUfuT4eGY5Tgo7smknTPr9guJZUx55D1VOR6\_Yw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8LJUcKUfuT4eGY5Tgo7smknTPr9guJZUx55D1VOR6_Yw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:36am UTC](https://discuss.elastic.co/t/importing-large-amounts-of-data-to-production-indices/21864/5 "2017-07-06T00:36:01Z")