# Geo\_shape indexing speed **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2013, 4:41pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861 "2013-07-19T16:41:02Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Mohamed\_Lrhazi](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mohamed\_Lrhazi](https://discuss.elastic.co/u/Mohamed_Lrhazi)\ **Post date:** [July 19, 2013, 4:41pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/1 "2013-07-19T16:41:02Z")
Hello, I need to index lots of documents in the following format: _source: { geobox: { type: envelope coordinates: [[-78.538,-0.363],[-78.53,-0.371]] } country: EC name: -0.363_-78.538\_-0.371\_-78.53 population: 231 } My type is like this: "cell" : { "properties" : { u"name": {'index': 'not\_analyzed','type': 'string',}, u"country": {'index': 'not\_analyzed','type': 'string',}, u"population": {'type': 'integer',}, u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1m"}, } } I tried many tuning suggestions I found, including increasing JVM RAM to 16GB, mlockall: true... None of my tuning changes seem to cause any dramatic improvement in indexing performance. I seem to always index 500 documents somewhere between 30 and 60 seconds.... am indexing on one VM, part of a 4 node cluster. index has 4 shards and zero replicas. Note that even going from one node to four did not change the indexing speed that much! am using Python and PyES (thrift) for indexing. Can I hope for much better performance with somehow? what would you suggest? or is this good enough for geo\_shape indexing? Thanks a lot, Mohamed. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [July 19, 2013, 4:50pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/2 "2013-07-19T16:50:05Z")
Hi Mohamed, How are you indexing those documents? Are you sending them using the bulk API? Are you using threads o processes to send more in paralel? Are you always talking to just one node? The fact that you don't see increased performance with more nodes might suggest that the bottleneck is actually your Python process. Try running the indexing in paralel to determine that. On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > Hello, > > I need to index lots of documents in the following format: > > _source: { > geobox: { > type: envelope > coordinates: [[-78.538,-0.363],[-78.53,-0.371]] > } > country: EC > name: -0.363_-78.538\_-0.371\_-78.53 > population: 231 > } > > My type is like this: > > "cell" : { > "properties" : { > u"name": {'index': 'not\_analyzed','type': 'string',}, > u"country": {'index': 'not\_analyzed','type': 'string',}, > u"population": {'type': 'integer',}, > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > "precision": "1m"}, > } > } > > I tried many tuning suggestions I found, including increasing JVM RAM to > 16GB, mlockall: true... > > None of my tuning changes seem to cause any dramatic improvement in > indexing performance. I seem to always index 500 documents somewhere > between 30 and 60 seconds.... > > am indexing on one VM, part of a 4 node cluster. index has 4 shards and > zero replicas. Note that even going from one node to four did not change > the indexing speed that much! > > am using Python and PyES (thrift) for indexing. > > Can I hope for much better performance with somehow? what would you > suggest? or is this good enough for geo\_shape indexing? > > Thanks a lot, > Mohamed. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohamed\_Lrhazi](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mohamed\_Lrhazi](https://discuss.elastic.co/u/Mohamed_Lrhazi)\ **Post date:** [July 19, 2013, 4:59pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/3 "2013-07-19T16:59:05Z")
Hi Honza, Yes, I am using the bulk API, with a bulk\_size of 5000. Tried other numbers but 5k seemed to give best results. I will rewrite my Python to use multi processing ans see if I can get more performance. and yes, I am always talking to one node only... I will test running four python instances one on each node. Mohamed. On Friday, July 19, 2013 12:50:05 PM UTC-4, Honza Král wrote: > Hi Mohamed, > > How are you indexing those documents? Are you sending them using the bulk > API? Are you using threads o processes to send more in paralel? Are you > always talking to just one node? > > The fact that you don't see increased performance with more nodes might > suggest that the bottleneck is actually your Python process. Try running > the indexing in paralel to determine that. > > On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi \ > > > wrote: > > > Hello, > > > > I need to index lots of documents in the following format: > > > > _source: { > > geobox: { > > type: envelope > > coordinates: [[-78.538,-0.363],[-78.53,-0.371]] > > } > > country: EC > > name: -0.363_-78.538\_-0.371\_-78.53 > > population: 231 > > } > > > > My type is like this: > > > > "cell" : { > > "properties" : { > > u"name": {'index': 'not\_analyzed','type': 'string',}, > > u"country": {'index': 'not\_analyzed','type': 'string',}, > > u"population": {'type': 'integer',}, > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > > "precision": "1m"}, > > } > > } > > > > I tried many tuning suggestions I found, including increasing JVM RAM to > > 16GB, mlockall: true... > > > > None of my tuning changes seem to cause any dramatic improvement in > > indexing performance. I seem to always index 500 documents somewhere > > between 30 and 60 seconds.... > > > > am indexing on one VM, part of a 4 node cluster. index has 4 shards and > > zero replicas. Note that even going from one node to four did not change > > the indexing speed that much! > > > > am using Python and PyES (thrift) for indexing. > > > > Can I hope for much better performance with somehow? what would you > > suggest? or is this good enough for geo\_shape indexing? > > > > Thanks a lot, > > Mohamed. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohamed\_Lrhazi](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mohamed\_Lrhazi](https://discuss.elastic.co/u/Mohamed_Lrhazi)\ **Post date:** [July 19, 2013, 6:46pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/4 "2013-07-19T18:46:11Z")
Using all four nodes for indexing, instead of one, improved the over all results, but not by much. Each node is taking somewhere between 60 and 100 seconds to index 5000 docs, instead of 30 to 60 seconds. In my first post I said 500 documents, but that was a mistake, it is 5,000, which is my PyES bulk\_size. Example script log: 2013/07/19 14:15:23: Checkpoint 45000 Time: [73.04 sec] 2013/07/19 14:16:35: Checkpoint 50000 Time: [69.91 sec] 2013/07/19 14:17:55: Checkpoint 55000 Time: [76.86 sec] 2013/07/19 14:19:14: Checkpoint 60000 Time: [76.40 sec] 2013/07/19 14:20:41: Checkpoint 65000 Time: [84.90 sec] 2013/07/19 14:21:59: Checkpoint 70000 Time: [74.73 sec] Do these numbers sound too low? Any other suggestions? Also, my Python script is simply reading already existing JSON strings off of file... Would a bash script using curl to push the documents to ES be faster? I have no idea where the bottleneck really is! Thanks a lot, Mohamed. On Fri, Jul 19, 2013 at 12:59 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > Hi Honza, > > Yes, I am using the bulk API, with a bulk\_size of 5000. Tried other > numbers but 5k seemed to give best results. > > I will rewrite my Python to use multi processing ans see if I can get more > performance. > > and yes, I am always talking to one node only... I will test running four > python instances one on each node. > > Mohamed. > > On Friday, July 19, 2013 12:50:05 PM UTC-4, Honza Král wrote: > > > Hi Mohamed, > > > > How are you indexing those documents? Are you sending them using the bulk > > API? Are you using threads o processes to send more in paralel? Are you > > always talking to just one node? > > > > The fact that you don't see increased performance with more nodes might > > suggest that the bottleneck is actually your Python process. Try running > > the indexing in paralel to determine that. > > > > On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi [ml...@georgetown.edu](mailto:ml...@georgetown.edu)wrote: > > > > > Hello, > > > > > > I need to index lots of documents in the following format: > > > > > > _source: { > > > geobox: { > > > type: envelope > > > coordinates: [[-78.538,-0.363],[-78.53,-0.\*\*371]] > > > } > > > country: EC > > > name: -0.363_-78.538\_-0.371\_-78.53 > > > population: 231 > > > } > > > > > > My type is like this: > > > > > > "cell" : { > > > "properties" : { > > > u"name": {'index': 'not\_analyzed','type': 'string',}, > > > u"country": {'index': 'not\_analyzed','type': 'string',}, > > > u"population": {'type': 'integer',}, > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > > > "precision": "1m"}, > > > } > > > } > > > > > > I tried many tuning suggestions I found, including increasing JVM RAM to > > > 16GB, mlockall: true... > > > > > > None of my tuning changes seem to cause any dramatic improvement in > > > indexing performance. I seem to always index 500 documents somewhere > > > between 30 and 60 seconds.... > > > > > > am indexing on one VM, part of a 4 node cluster. index has 4 shards and > > > zero replicas. Note that even going from one node to four did not change > > > the indexing speed that much! > > > > > > am using Python and PyES (thrift) for indexing. > > > > > > Can I hope for much better performance with somehow? what would you > > > suggest? or is this good enough for geo\_shape indexing? > > > > > > Thanks a lot, > > > Mohamed. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohamed\_Lrhazi](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mohamed\_Lrhazi](https://discuss.elastic.co/u/Mohamed_Lrhazi)\ **Post date:** [July 20, 2013, 5:33pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/5 "2013-07-20T17:33:14Z")
Could this change dramatically effect indexing speed AND index size? u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1m"}, to: u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1km"}, am trying it now, plus running 10 concurrent curl POSTs, and the process is moving very fast (sorry, not collecting any metrics)... Thanks, Mohamed. On Fri, Jul 19, 2013 at 2:46 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > Using all four nodes for indexing, instead of one, improved the over all > results, but not by much. Each node is taking somewhere between 60 and 100 > seconds to index 5000 docs, instead of 30 to 60 seconds. > In my first post I said 500 documents, but that was a mistake, it is > 5,000, which is my PyES bulk\_size. > > Example script log: > > 2013/07/19 14:15:23: Checkpoint 45000 Time: [73.04 sec] > 2013/07/19 14:16:35: Checkpoint 50000 Time: [69.91 sec] > 2013/07/19 14:17:55: Checkpoint 55000 Time: [76.86 sec] > 2013/07/19 14:19:14: Checkpoint 60000 Time: [76.40 sec] > 2013/07/19 14:20:41: Checkpoint 65000 Time: [84.90 sec] > 2013/07/19 14:21:59: Checkpoint 70000 Time: [74.73 sec] > > Do these numbers sound too low? Any other suggestions? > > Also, my Python script is simply reading already existing JSON strings off > of file... Would a bash script using curl to push the documents to ES be > faster? I have no idea where the bottleneck really is! > > Thanks a lot, > Mohamed. > > On Fri, Jul 19, 2013 at 12:59 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > Hi Honza, > > > > Yes, I am using the bulk API, with a bulk\_size of 5000. Tried other > > numbers but 5k seemed to give best results. > > > > I will rewrite my Python to use multi processing ans see if I can get > > more performance. > > > > and yes, I am always talking to one node only... I will test running four > > python instances one on each node. > > > > Mohamed. > > > > On Friday, July 19, 2013 12:50:05 PM UTC-4, Honza Král wrote: > > > > > Hi Mohamed, > > > > > > How are you indexing those documents? Are you sending them using the > > > bulk API? Are you using threads o processes to send more in paralel? Are > > > you always talking to just one node? > > > > > > The fact that you don't see increased performance with more nodes might > > > suggest that the bottleneck is actually your Python process. Try running > > > the indexing in paralel to determine that. > > > > > > On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi [ml...@georgetown.edu](mailto:ml...@georgetown.edu)wrote: > > > > > > > Hello, > > > > > > > > I need to index lots of documents in the following format: > > > > > > > > _source: { > > > > geobox: { > > > > type: envelope > > > > coordinates: [[-78.538,-0.363],[-78.53,-0.\*\*371]] > > > > } > > > > country: EC > > > > name: -0.363_-78.538\_-0.371\_-78.53 > > > > population: 231 > > > > } > > > > > > > > My type is like this: > > > > > > > > "cell" : { > > > > "properties" : { > > > > u"name": {'index': 'not\_analyzed','type': 'string',}, > > > > u"country": {'index': 'not\_analyzed','type': 'string',}, > > > > u"population": {'type': 'integer',}, > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > > > > "precision": "1m"}, > > > > } > > > > } > > > > > > > > I tried many tuning suggestions I found, including increasing JVM RAM > > > > to 16GB, mlockall: true... > > > > > > > > None of my tuning changes seem to cause any dramatic improvement in > > > > indexing performance. I seem to always index 500 documents somewhere > > > > between 30 and 60 seconds.... > > > > > > > > am indexing on one VM, part of a 4 node cluster. index has 4 shards and > > > > zero replicas. Note that even going from one node to four did not change > > > > the indexing speed that much! > > > > > > > > am using Python and PyES (thrift) for indexing. > > > > > > > > Can I hope for much better performance with somehow? what would you > > > > suggest? or is this good enough for geo\_shape indexing? > > > > > > > > Thanks a lot, > > > > Mohamed. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit > > > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe). > > > To unsubscribe from this group and all its topics, send an email to > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Honza\_Kral](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/honza_kral/32/876_2.png) [@Honza\_Kral](https://discuss.elastic.co/u/Honza_Kral)\ **Post date:** [July 20, 2013, 5:34pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/6 "2013-07-20T17:34:57Z")
Absolutely, the precision has a huge impact on the amount of work es needs to do to the data, lower precision reduces the cpu load significantly On Sat, Jul 20, 2013 at 7:33 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > Could this change dramatically effect indexing speed AND index size? > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1m"}, > > to: > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1km"}, > > am trying it now, plus running 10 concurrent curl POSTs, and the process > is moving very fast (sorry, not collecting any metrics)... > > Thanks, > Mohamed. > > On Fri, Jul 19, 2013 at 2:46 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > Using all four nodes for indexing, instead of one, improved the over all > > results, but not by much. Each node is taking somewhere between 60 and 100 > > seconds to index 5000 docs, instead of 30 to 60 seconds. > > In my first post I said 500 documents, but that was a mistake, it is > > 5,000, which is my PyES bulk\_size. > > > > Example script log: > > > > 2013/07/19 14:15:23: Checkpoint 45000 Time: [73.04 sec] > > 2013/07/19 14:16:35: Checkpoint 50000 Time: [69.91 sec] > > 2013/07/19 14:17:55: Checkpoint 55000 Time: [76.86 sec] > > 2013/07/19 14:19:14: Checkpoint 60000 Time: [76.40 sec] > > 2013/07/19 14:20:41: Checkpoint 65000 Time: [84.90 sec] > > 2013/07/19 14:21:59: Checkpoint 70000 Time: [74.73 sec] > > > > Do these numbers sound too low? Any other suggestions? > > > > Also, my Python script is simply reading already existing JSON strings > > off of file... Would a bash script using curl to push the documents to ES > > be faster? I have no idea where the bottleneck really is! > > > > Thanks a lot, > > Mohamed. > > > > On Fri, Jul 19, 2013 at 12:59 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > > > Hi Honza, > > > > > > Yes, I am using the bulk API, with a bulk\_size of 5000. Tried other > > > numbers but 5k seemed to give best results. > > > > > > I will rewrite my Python to use multi processing ans see if I can get > > > more performance. > > > > > > and yes, I am always talking to one node only... I will test running > > > four python instances one on each node. > > > > > > Mohamed. > > > > > > On Friday, July 19, 2013 12:50:05 PM UTC-4, Honza Král wrote: > > > > > > > Hi Mohamed, > > > > > > > > How are you indexing those documents? Are you sending them using the > > > > bulk API? Are you using threads o processes to send more in paralel? Are > > > > you always talking to just one node? > > > > > > > > The fact that you don't see increased performance with more nodes might > > > > suggest that the bottleneck is actually your Python process. Try running > > > > the indexing in paralel to determine that. > > > > > > > > On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi [ml...@georgetown.edu](mailto:ml...@georgetown.edu)wrote: > > > > > > > > > Hello, > > > > > > > > > > I need to index lots of documents in the following format: > > > > > > > > > > _source: { > > > > > geobox: { > > > > > type: envelope > > > > > coordinates: [[-78.538,-0.363],[-78.53,-0.\*\*371]] > > > > > } > > > > > country: EC > > > > > name: -0.363_-78.538\_-0.371\_-78.53 > > > > > population: 231 > > > > > } > > > > > > > > > > My type is like this: > > > > > > > > > > "cell" : { > > > > > "properties" : { > > > > > u"name": {'index': 'not\_analyzed','type': 'string',}, > > > > > u"country": {'index': 'not\_analyzed','type': 'string',}, > > > > > u"population": {'type': 'integer',}, > > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > > > > > "precision": "1m"}, > > > > > } > > > > > } > > > > > > > > > > I tried many tuning suggestions I found, including increasing JVM RAM > > > > > to 16GB, mlockall: true... > > > > > > > > > > None of my tuning changes seem to cause any dramatic improvement in > > > > > indexing performance. I seem to always index 500 documents somewhere > > > > > between 30 and 60 seconds.... > > > > > > > > > > am indexing on one VM, part of a 4 node cluster. index has 4 shards > > > > > and zero replicas. Note that even going from one node to four did not > > > > > change the indexing speed that much! > > > > > > > > > > am using Python and PyES (thrift) for indexing. > > > > > > > > > > Can I hope for much better performance with somehow? what would you > > > > > suggest? or is this good enough for geo\_shape indexing? > > > > > > > > > > Thanks a lot, > > > > > Mohamed. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > > > -- > > > > You received this message because you are subscribed to a topic in the > > > > Google Groups "elasticsearch" group. > > > > To unsubscribe from this topic, visit > > > > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe). > > > > To unsubscribe from this group and all its topics, send an email to > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohamed\_Lrhazi](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mohamed\_Lrhazi](https://discuss.elastic.co/u/Mohamed_Lrhazi)\ **Post date:** [July 20, 2013, 5:38pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/7 "2013-07-20T17:38:33Z")
Excellent, thanks a lot for confirming. I think 1km precision will do just fine for my needs. Mohamed. On Sat, Jul 20, 2013 at 1:34 PM, Honza Král [honza.kral@elasticsearch.com](mailto:honza.kral@elasticsearch.com)wrote: > Absolutely, the precision has a huge impact on the amount of work es needs > to do to the data, lower precision reduces the cpu load significantly > > On Sat, Jul 20, 2013 at 7:33 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > Could this change dramatically effect indexing speed AND index size? > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1m"}, > > > > to: > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1km"}, > > > > am trying it now, plus running 10 concurrent curl POSTs, and the process > > is moving very fast (sorry, not collecting any metrics)... > > > > Thanks, > > Mohamed. > > > > On Fri, Jul 19, 2013 at 2:46 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > > > Using all four nodes for indexing, instead of one, improved the over all > > > results, but not by much. Each node is taking somewhere between 60 and 100 > > > seconds to index 5000 docs, instead of 30 to 60 seconds. > > > In my first post I said 500 documents, but that was a mistake, it is > > > 5,000, which is my PyES bulk\_size. > > > > > > Example script log: > > > > > > 2013/07/19 14:15:23: Checkpoint 45000 Time: [73.04 sec] > > > 2013/07/19 14:16:35: Checkpoint 50000 Time: [69.91 sec] > > > 2013/07/19 14:17:55: Checkpoint 55000 Time: [76.86 sec] > > > 2013/07/19 14:19:14: Checkpoint 60000 Time: [76.40 sec] > > > 2013/07/19 14:20:41: Checkpoint 65000 Time: [84.90 sec] > > > 2013/07/19 14:21:59: Checkpoint 70000 Time: [74.73 sec] > > > > > > Do these numbers sound too low? Any other suggestions? > > > > > > Also, my Python script is simply reading already existing JSON strings > > > off of file... Would a bash script using curl to push the documents to ES > > > be faster? I have no idea where the bottleneck really is! > > > > > > Thanks a lot, > > > Mohamed. > > > > > > On Fri, Jul 19, 2013 at 12:59 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > > > > > Hi Honza, > > > > > > > > Yes, I am using the bulk API, with a bulk\_size of 5000. Tried other > > > > numbers but 5k seemed to give best results. > > > > > > > > I will rewrite my Python to use multi processing ans see if I can get > > > > more performance. > > > > > > > > and yes, I am always talking to one node only... I will test running > > > > four python instances one on each node. > > > > > > > > Mohamed. > > > > > > > > On Friday, July 19, 2013 12:50:05 PM UTC-4, Honza Král wrote: > > > > > > > > > Hi Mohamed, > > > > > > > > > > How are you indexing those documents? Are you sending them using the > > > > > bulk API? Are you using threads o processes to send more in paralel? Are > > > > > you always talking to just one node? > > > > > > > > > > The fact that you don't see increased performance with more nodes > > > > > might suggest that the bottleneck is actually your Python process. Try > > > > > running the indexing in paralel to determine that. > > > > > > > > > > On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi [ml...@georgetown.edu](mailto:ml...@georgetown.edu)wrote: > > > > > > > > > > > Hello, > > > > > > > > > > > > I need to index lots of documents in the following format: > > > > > > > > > > > > _source: { > > > > > > geobox: { > > > > > > type: envelope > > > > > > coordinates: [[-78.538,-0.363],[-78.53,-0.\*\*371]] > > > > > > } > > > > > > country: EC > > > > > > name: -0.363_-78.538\_-0.371\_-78.53 > > > > > > population: 231 > > > > > > } > > > > > > > > > > > > My type is like this: > > > > > > > > > > > > "cell" : { > > > > > > "properties" : { > > > > > > u"name": {'index': 'not\_analyzed','type': 'string',}, > > > > > > u"country": {'index': 'not\_analyzed','type': 'string',}, > > > > > > u"population": {'type': 'integer',}, > > > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > > > > > > "precision": "1m"}, > > > > > > } > > > > > > } > > > > > > > > > > > > I tried many tuning suggestions I found, including increasing JVM RAM > > > > > > to 16GB, mlockall: true... > > > > > > > > > > > > None of my tuning changes seem to cause any dramatic improvement in > > > > > > indexing performance. I seem to always index 500 documents somewhere > > > > > > between 30 and 60 seconds.... > > > > > > > > > > > > am indexing on one VM, part of a 4 node cluster. index has 4 shards > > > > > > and zero replicas. Note that even going from one node to four did not > > > > > > change the indexing speed that much! > > > > > > > > > > > > am using Python and PyES (thrift) for indexing. > > > > > > > > > > > > Can I hope for much better performance with somehow? what would you > > > > > > suggest? or is this good enough for geo\_shape indexing? > > > > > > > > > > > > Thanks a lot, > > > > > > Mohamed. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > . > > > > > > > > > > -- > > > > > You received this message because you are subscribed to a topic in the > > > > > Google Groups "elasticsearch" group. > > > > > To unsubscribe from this topic, visit > > > > > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe) > > > > > . > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Mohamed\_Lrhazi](https://avatars.discourse-cdn.com/v4/letter/m/74df32/32.png) [@Mohamed\_Lrhazi](https://discuss.elastic.co/u/Mohamed_Lrhazi)\ **Post date:** [July 20, 2013, 10:45pm UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/8 "2013-07-20T22:45:07Z")
For the record: Indexed 75 million such documents in about 4 hours. Used some 65GB disk space. Used 10 concurrent curl commands on one node, part of a four node cluster. Cluster nodes happened to be 4 big VMs: two of them had 16 CPUs and 200GB RAM, two had 8 CPUs and 50 GB RAM. JVM configured on all four nodes with max RAM of 16GB. On Sat, Jul 20, 2013 at 1:38 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > Excellent, thanks a lot for confirming. I think 1km precision will do just > fine for my needs. > > Mohamed. > > On Sat, Jul 20, 2013 at 1:34 PM, Honza Král [honza.kral@elasticsearch.com](mailto:honza.kral@elasticsearch.com)wrote: > > > Absolutely, the precision has a huge impact on the amount of work es > > needs to do to the data, lower precision reduces the cpu load significantly > > > > On Sat, Jul 20, 2013 at 7:33 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > > > Could this change dramatically effect indexing speed AND index size? > > > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1m"}, > > > > > > to: > > > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", "precision": "1km"}, > > > > > > am trying it now, plus running 10 concurrent curl POSTs, and the process > > > is moving very fast (sorry, not collecting any metrics)... > > > > > > Thanks, > > > Mohamed. > > > > > > On Fri, Jul 19, 2013 at 2:46 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > > > > > Using all four nodes for indexing, instead of one, improved the over > > > > all results, but not by much. Each node is taking somewhere between 60 and > > > > 100 seconds to index 5000 docs, instead of 30 to 60 seconds. > > > > In my first post I said 500 documents, but that was a mistake, it is > > > > 5,000, which is my PyES bulk\_size. > > > > > > > > Example script log: > > > > > > > > 2013/07/19 14:15:23: Checkpoint 45000 Time: [73.04 sec] > > > > 2013/07/19 14:16:35: Checkpoint 50000 Time: [69.91 sec] > > > > 2013/07/19 14:17:55: Checkpoint 55000 Time: [76.86 sec] > > > > 2013/07/19 14:19:14: Checkpoint 60000 Time: [76.40 sec] > > > > 2013/07/19 14:20:41: Checkpoint 65000 Time: [84.90 sec] > > > > 2013/07/19 14:21:59: Checkpoint 70000 Time: [74.73 sec] > > > > > > > > Do these numbers sound too low? Any other suggestions? > > > > > > > > Also, my Python script is simply reading already existing JSON strings > > > > off of file... Would a bash script using curl to push the documents to ES > > > > be faster? I have no idea where the bottleneck really is! > > > > > > > > Thanks a lot, > > > > Mohamed. > > > > > > > > On Fri, Jul 19, 2013 at 12:59 PM, Mohamed Lrhazi [ml623@georgetown.edu](mailto:ml623@georgetown.edu)wrote: > > > > > > > > > Hi Honza, > > > > > > > > > > Yes, I am using the bulk API, with a bulk\_size of 5000. Tried other > > > > > numbers but 5k seemed to give best results. > > > > > > > > > > I will rewrite my Python to use multi processing ans see if I can get > > > > > more performance. > > > > > > > > > > and yes, I am always talking to one node only... I will test running > > > > > four python instances one on each node. > > > > > > > > > > Mohamed. > > > > > > > > > > On Friday, July 19, 2013 12:50:05 PM UTC-4, Honza Král wrote: > > > > > > > > > > > Hi Mohamed, > > > > > > > > > > > > How are you indexing those documents? Are you sending them using the > > > > > > bulk API? Are you using threads o processes to send more in paralel? Are > > > > > > you always talking to just one node? > > > > > > > > > > > > The fact that you don't see increased performance with more nodes > > > > > > might suggest that the bottleneck is actually your Python process. Try > > > > > > running the indexing in paralel to determine that. > > > > > > > > > > > > On Fri, Jul 19, 2013 at 6:41 PM, Mohamed Lrhazi \ > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hello, > > > > > > > > > > > > > > I need to index lots of documents in the following format: > > > > > > > > > > > > > > _source: { > > > > > > > geobox: { > > > > > > > type: envelope > > > > > > > coordinates: [[-78.538,-0.363],[-78.53,-0.\*\*371]] > > > > > > > } > > > > > > > country: EC > > > > > > > name: -0.363_-78.538\_-0.371\_-78.53 > > > > > > > population: 231 > > > > > > > } > > > > > > > > > > > > > > My type is like this: > > > > > > > > > > > > > > "cell" : { > > > > > > > "properties" : { > > > > > > > u"name": {'index': 'not\_analyzed','type': 'string',}, > > > > > > > u"country": {'index': 'not\_analyzed','type': 'string',}, > > > > > > > u"population": {'type': 'integer',}, > > > > > > > u"geobox": {'type': 'geo\_shape', "tree": "geohash", > > > > > > > "precision": "1m"}, > > > > > > > } > > > > > > > } > > > > > > > > > > > > > > I tried many tuning suggestions I found, including increasing JVM > > > > > > > RAM to 16GB, mlockall: true... > > > > > > > > > > > > > > None of my tuning changes seem to cause any dramatic improvement in > > > > > > > indexing performance. I seem to always index 500 documents somewhere > > > > > > > between 30 and 60 seconds.... > > > > > > > > > > > > > > am indexing on one VM, part of a 4 node cluster. index has 4 shards > > > > > > > and zero replicas. Note that even going from one node to four did not > > > > > > > change the indexing speed that much! > > > > > > > > > > > > > > am using Python and PyES (thrift) for indexing. > > > > > > > > > > > > > > Can I hope for much better performance with somehow? what would you > > > > > > > suggest? or is this good enough for geo\_shape indexing? > > > > > > > > > > > > > > Thanks a lot, > > > > > > > Mohamed. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > . > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to a topic in the > > > > > > Google Groups "elasticsearch" group. > > > > > > To unsubscribe from this topic, visit > > > > > > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe) > > > > > > . > > > > > > To unsubscribe from this group and all its topics, send an email to > > > > > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe](https://groups.google.com/d/topic/elasticsearch/OXJVD-9HtUs/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:25am UTC](https://discuss.elastic.co/t/geo-shape-indexing-speed/12861/9 "2017-07-06T02:25:22Z")