# Rails: Tire import taking too long **URL:** \ **Category:** Elasticsearch\ **Created:** [October 10, 2013, 1:37am UTC](https://discuss.elastic.co/t/rails-tire-import-taking-too-long/13897 "2013-10-10T01:37:12Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Karan\_Verma](https://avatars.discourse-cdn.com/v4/letter/k/858c86/32.png) [@Karan\_Verma](https://discuss.elastic.co/u/Karan_Verma)\ **Post date:** [October 10, 2013, 1:37am UTC](https://discuss.elastic.co/t/rails-tire-import-taking-too-long/13897/1 "2013-10-10T01:37:12Z")
Hi I want to index close to 900,000 documents but it is taking quite long. I have defined a mapping which contains indexes is on the lines of: ``` indexes :residencies_with_year, type: 'string', :as => ``` 'residencies\_obj.map{|r| ExpertProfile.residency\_with\_year\_to\_s(r)}' Thus I believe there are quite a few database queries as I'm indexing the model. I'm using the default configuration number of nodes =1, number of shards=5, and number of replicas=1. Each batch of 1000 documents is taking about 15 minutes. How can I speed it up? rake environment tire:import CLASS='Expert' FORCE=true -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [October 10, 2013, 9:02am UTC](https://discuss.elastic.co/t/rails-tire-import-taking-too-long/13897/2 "2013-10-10T09:02:17Z")
Hi, I've got couple of questions: 1/ What kind of database are you reading the data from (SQLite, PostgreSQL, etc)? Locally or in production? 2/ Assuming you're using ActiveRecord, can you enable logging and have a look if you're not hit with n+1 queries? Just put something like this in your Rakefile or the Tire initializer? ActiveRecord::Base.logger = Logger.new(STDOUT) ActiveRecord::Base.clear\_active\_connections! Indexing a 1,000 documents should take seconds at most, usually. Karel On Thursday, October 10, 2013 3:37:12 AM UTC+2, Karan Verma wrote: > Hi > > I want to index close to 900,000 documents but it is taking quite long. I > have defined a mapping which contains indexes is on the lines of: > > ``` > indexes :residencies_with_year, type: 'string', :as => > > ``` > > 'residencies\_obj.map{|r| ExpertProfile.residency\_with\_year\_to\_s(r)}' > > Thus I believe there are quite a few database queries as I'm indexing the > model. I'm using the default configuration number of nodes =1, number of > shards=5, and number of replicas=1. > Each batch of 1000 documents is taking about 15 minutes. > > How can I speed it up? > > rake environment tire:import CLASS='Expert' FORCE=true -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Karan\_Verma](https://avatars.discourse-cdn.com/v4/letter/k/858c86/32.png) [@Karan\_Verma](https://discuss.elastic.co/u/Karan_Verma)\ **Post date:** [October 10, 2013, 7:49pm UTC](https://discuss.elastic.co/t/rails-tire-import-taking-too-long/13897/3 "2013-10-10T19:49:53Z")
Hi Karel 1. Data is being read from MySQL and DynamoDB and both are remote environments. 2. Initially i was creating the index on ES server running on localhost. I setup one on a separate EC2 instance. I enabled logging and the time it takes to load one Expert from the model is about 74ms. Thus for 1000 documents it should take 1.5 minutes but its taking 3 minutes which is still better. I checked for the n+1 issue and there are some optimizations possible there, but I guess 74ms for one record is not that bad? However for 900,000 this still seems it'll take quite some time. Is there a way to index in parallel? How about: rake environment tire:import CLASS="Expert.where('id % 10 = 9')" without the Force = true option, which I guess, wouldn't cause the index to be deleted and I could run 10 rake tasks in parallel. Would this cause any problems? Is there a better way? On Thu, Oct 10, 2013 at 2:02 AM, Karel Minařík [karel.minarik@gmail.com](mailto:karel.minarik@gmail.com)wrote: > Hi, > > I've got couple of questions: > > 1/ What kind of database are you reading the data from (SQLite, > PostgreSQL, etc)? Locally or in production? > > 2/ Assuming you're using ActiveRecord, can you enable logging and have a > look if you're not hit with n+1 queries? Just put something like this in > your Rakefile or the Tire initializer? > > ActiveRecord::Base.logger = Logger.new(STDOUT) > ActiveRecord::Base.clear\_active\_connections! > > Indexing a 1,000 documents should take seconds at most, usually. > > Karel > > On Thursday, October 10, 2013 3:37:12 AM UTC+2, Karan Verma wrote: > > > Hi > > > > I want to index close to 900,000 documents but it is taking quite long. I > > have defined a mapping which contains indexes is on the lines of: > > > > ``` > > indexes :residencies_with_year, type: 'string', :as => > > > > ``` > > > > 'residencies\_obj.map{|r| ExpertProfile.residency\_with\_\*\*year\_to\_s(r)}' > > > > Thus I believe there are quite a few database queries as I'm indexing the > > model. I'm using the default configuration number of nodes =1, number of > > shards=5, and number of replicas=1. > > Each batch of 1000 documents is taking about 15 minutes. > > > > How can I speed it up? > > > > rake environment tire:import CLASS='Expert' FORCE=true > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/pvEl7nwK5cg/unsubscribe](https://groups.google.com/d/topic/elasticsearch/pvEl7nwK5cg/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Best, Karan Life saving Ninja & Software Engineer Karan pronounced Ka ([http://tiny.cc/0lu61w](http://tiny.cc/0lu61w)) + Run -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Karan\_Verma](https://avatars.discourse-cdn.com/v4/letter/k/858c86/32.png) [@Karan\_Verma](https://discuss.elastic.co/u/Karan_Verma)\ **Post date:** [October 10, 2013, 7:54pm UTC](https://discuss.elastic.co/t/rails-tire-import-taking-too-long/13897/4 "2013-10-10T19:54:49Z")
Update: The above method doesn't seem to work. Including the trace. * * * 1000/98754 | 1% rake aborted! The level of configured provisioned throughput for the table was exceeded. Consider increasing your provisioning level with the UpdateTable API /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/model/search.rb:184:in `instance_eval' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.5.8/lib/aws/core/client.rb:376:in `client\_request' (eval):3:in `get_item' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.5.8/lib/aws/dynamo_db/attribute_collection.rb:437:in `get\_item' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/aws-sdk-1.5.8/lib/aws/dynamo\_db/attribute\_collection.rb:426:in `to_h' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/activesupport-3.1.12/lib/active_support/core_ext/object/try.rb:32:in `try' /Users/karan/ht-webapp12/app/models/expert/nosql.rb:141:in `get_expert_profile_nosql' /Users/karan/ht-webapp12/app/models/expert/nosql.rb:71:in `specialties' /Users/karan/ht-webapp12/app/models/expert.rb:3360:in `specialty_names' /Users/karan/ht-webapp12/app/models/expert.rb:686:in `relevancy\_tags' (eval):1:in `block in to_indexed_json' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/model/search.rb:184:in `instance\_eval' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/model/search.rb:184:in `block in to_indexed_json' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/model/search.rb:181:in `each' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/model/search.rb:181:in `to_indexed_json' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/model/search.rb:312:in `to\_indexed\_json' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/index.rb:432:in `convert_document_to_json' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/index.rb:138:in `block in bulk' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/activerecord-3.1.12/lib/active\_record/relation.rb:15:in `map' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/activerecord-3.1.12/lib/active_record/relation.rb:15:in `map' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/index.rb:114:in `bulk' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/index.rb:178:in `bulk\_store' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/index.rb:194:in `import' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/tasks.rb:86:in `block (3 levels) in \' /Users/karan/.rvm/gems/ruby-1.9.3-p448/gems/tire-0.5.3/lib/tire/tasks.rb:72:in `block (2 levels) in ' /Users/karan/.rvm/gems/ruby-1.9.3-p448/bin/ruby_executable_hooks:15:in `eval' /Users/karan/.rvm/gems/ruby-1.9.3-p448/bin/ruby\_executable\_hooks:15:in `' Tasks: TOP =\> tire:import (See full trace by running task with --trace) On Thu, Oct 10, 2013 at 12:49 PM, Karan Verma [karan@healthtap.com](mailto:karan@healthtap.com) wrote: > Hi Karel > > 1. Data is being read from MySQL and DynamoDB and both are remote > environments. > > 2. Initially i was creating the index on ES server running on localhost. I > setup one on a separate EC2 instance. I enabled logging and the time it > takes to load one Expert from the model is about 74ms. Thus for 1000 > documents it should take 1.5 minutes but its taking 3 minutes which is > still better. I checked for the n+1 issue and there are some optimizations > possible there, but I guess 74ms for one record is not that bad? > > However for 900,000 this still seems it'll take quite some time. Is there > a way to index in parallel? > > How about: > > rake environment tire:import CLASS="Expert.where('id % 10 = 9')" > > without the Force = true option, which I guess, wouldn't cause the index > to be deleted and I could run 10 rake tasks in parallel. Would this cause > any problems? Is there a better way? > > On Thu, Oct 10, 2013 at 2:02 AM, Karel Minařík [karel.minarik@gmail.com](mailto:karel.minarik@gmail.com)wrote: > > > Hi, > > > > I've got couple of questions: > > > > 1/ What kind of database are you reading the data from (SQLite, > > PostgreSQL, etc)? Locally or in production? > > > > 2/ Assuming you're using ActiveRecord, can you enable logging and have a > > look if you're not hit with n+1 queries? Just put something like this in > > your Rakefile or the Tire initializer? > > > > ActiveRecord::Base.logger = Logger.new(STDOUT) > > ActiveRecord::Base.clear\_active\_connections! > > > > Indexing a 1,000 documents should take seconds at most, usually. > > > > Karel > > > > On Thursday, October 10, 2013 3:37:12 AM UTC+2, Karan Verma wrote: > > > > > Hi > > > > > > I want to index close to 900,000 documents but it is taking quite long. > > > I have defined a mapping which contains indexes is on the lines of: > > > > > > ``` > > > indexes :residencies_with_year, type: 'string', :as => > > > > > > ``` > > > > > > 'residencies\_obj.map{|r| ExpertProfile.residency\_with\_\*\*year\_to\_s(r)}' > > > > > > Thus I believe there are quite a few database queries as I'm indexing > > > the model. I'm using the default configuration number of nodes =1, number > > > of shards=5, and number of replicas=1. > > > Each batch of 1000 documents is taking about 15 minutes. > > > > > > How can I speed it up? > > > > > > rake environment tire:import CLASS='Expert' FORCE=true > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/pvEl7nwK5cg/unsubscribe](https://groups.google.com/d/topic/elasticsearch/pvEl7nwK5cg/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Best, > Karan > > Life saving Ninja & Software Engineer > > Karan pronounced Ka ([http://tiny.cc/0lu61w](http://tiny.cc/0lu61w)) + Run -- Best, Karan Life saving Ninja & Software Engineer Karan pronounced Ka ([http://tiny.cc/0lu61w](http://tiny.cc/0lu61w)) + Run -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:12am UTC](https://discuss.elastic.co/t/rails-tire-import-taking-too-long/13897/5 "2017-07-06T02:12:58Z")