# Bulk indexing tips for Elastic search and Cassandra River **URL:** \ **Category:** Elasticsearch\ **Created:** [March 29, 2013, 9:01pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387 "2013-03-29T21:01:14Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![utkarsh2012](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/utkarsh2012/32/1187_2.png) [@utkarsh2012](https://discuss.elastic.co/u/utkarsh2012)\ **Post date:** [March 29, 2013, 9:01pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/1 "2013-03-29T21:01:14Z")
Hello, I have been working on a cassandra river which triggers periodically and indexes all data in a cassandra column family. The implementation for now spawns 10 threads and processes 10k documents (with 13 columns)/thread. The performance initially was very good. It indexed 1M documents in 10mins. But after a 1hour, the indexing became very slow and it indexed around 8M documents. I am trying to index a total of 50M documents. I have attached a screenshot of the memory and CPU usage. What I noticed was, a lot of merge threads spawned up which reduced the speed considerably: "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] So, I believe this has to do with some configuration which I can tweak to improve bulk indexing. I am running 1 node with 5 shared with 2GB of ES\_HEAP\_SIZE and no replicas for now. Shay mentioned some tips here: [https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU) in 2011. Wanted to know if there are any bulk indexing performance improvements? I am also using: bulk.execute().addListener() (async) in place of bulk.execute().actionGet() (sync) I am planning to share the cassandra-river as soon its achieves acceptable performance. [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) Thanks, -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 29, 2013, 10:54pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/2 "2013-03-29T22:54:11Z")
You need to throttle bulk indexing, for example with org.elasticsearch.action.bulk.BulkProcessor, in order to get stable feed rate over long periods of time. Check out JDBC river for an example: [https://github.com/jprante/elasticsearch-river-jdbc/blob/master/src/main/java/org/elasticsearch/river/jdbc/strategy/simple/SimpleRiverMouth.java](https://github.com/jprante/elasticsearch-river-jdbc/blob/master/src/main/java/org/elasticsearch/river/jdbc/strategy/simple/SimpleRiverMouth.java) What is your document size? It is also valuable to measure bulk indexing throughput in MB/sec. I can index up to 8 MB/sec on an average desktop PC in a single node cluster (which is ~10-15.000 docs per sec for my doc size). Note, indexing speed can greatly be improved by fixing the slowest part of the system first: the I/O subsystem. Fast disk arrays (RAID 0) or even SSD devices help a lot. Jörg Am 29.03.13 22:01, schrieb [utkarsh2012@gmail.com](mailto:utkarsh2012@gmail.com): > Hello, > > I have been working on a cassandra river which triggers periodically > and indexes all data in a cassandra column family. The implementation > for now spawns 10 threads and processes 10k documents (with 13 > columns)/thread. > The performance initially was very good. It indexed 1M documents in > 10mins. But after a 1hour, the indexing became very slow and it > indexed around 8M documents. I am trying to index a total of 50M > documents. > > I have attached a screenshot of the memory and CPU usage. What I > noticed was, a lot of merge threads spawned up which reduced the speed > considerably: > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" > daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > So, I believe this has to do with some configuration which I can tweak > to improve bulk indexing. I am running 1 node with 5 shared with 2GB > of ES\_HEAP\_SIZE and no replicas for now. > > Shay mentioned some tips here: > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU) > in 2011. > Wanted to know if there are any bulk indexing performance improvements? > > I am also using: bulk.execute().addListener() (async) in place of > bulk.execute().actionGet() (sync) > > I am planning to share the cassandra-river as soon its achieves > acceptable performance. > > [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) > > [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) > > ## Thanks, -Utkarsh > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ashwini](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@ashwini](https://discuss.elastic.co/u/ashwini)\ **Post date:** [November 19, 2013, 12:55am UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/3 "2013-11-19T00:55:45Z")
I am new to elasticsearch and trying to learn how to integrate elasticsearch with cassandra. Can you give me some pointers. Thanks, Ashwini ---
**Author:** ![ashwini](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@ashwini](https://discuss.elastic.co/u/ashwini)\ **Post date:** [November 19, 2013, 10:41pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/4 "2013-11-19T22:41:23Z")
Hello, I am new to elasticsearch. I am trying use [GitHub - eBay/cassandra-river: Cassandra river for Elastic search.](https://github.com/eBay/cassandra-river) plugin to store data to cassandra. I dont see the data in cassandra. My search queries are returning from the elasticsearch. Please advise. Thanks, Ashwini On Friday, March 29, 2013 2:01:14 PM UTC-7, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > Hello, > > I have been working on a cassandra river which triggers periodically and > indexes all data in a cassandra column family. The implementation for now > spawns 10 threads and processes 10k documents (with 13 columns)/thread. > The performance initially was very good. It indexed 1M documents in > 10mins. But after a 1hour, the indexing became very slow and it indexed > around 8M documents. I am trying to index a total of 50M documents. > > I have attached a screenshot of the memory and CPU usage. What I noticed > was, a lot of merge threads spawned up which reduced the speed considerably: > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" > daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > So, I believe this has to do with some configuration which I can tweak to > improve bulk indexing. I am running 1 node with 5 shared with 2GB of > ES\_HEAP\_SIZE and no replicas for now. > > Shay mentioned some tips here: > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeUin) 2011. > Wanted to know if there are any bulk indexing performance improvements? > > I am also using: bulk.execute().addListener() (async) in place of > bulk.execute().actionGet() (sync) > > I am planning to share the cassandra-river as soon its achieves acceptable > performance. > > [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) > > [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) > > Thanks, > -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![utkarsh2012](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/utkarsh2012/32/1187_2.png) [@utkarsh2012](https://discuss.elastic.co/u/utkarsh2012)\ **Post date:** [November 19, 2013, 10:57pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/5 "2013-11-19T22:57:07Z")
Hi Ashwini, A river works the other way around, it moves data from cassandra to elasticsearch so that you can index it. I wrote cassandra-river as a proof of concept and hacked up in few days. Please feel free to file a bug or send a pull request. > **[Issues · eBay/cassandra-river](https://github.com/eBay/cassandra-river/issues)** > > Cassandra river for Elastic search. Contribute to eBay/cassandra-river development by creating an account on GitHub. Thanks, -Utkarsh On Tue, Nov 19, 2013 at 2:41 PM, Ashwini Lohit [ashwini.2284@gmail.com](mailto:ashwini.2284@gmail.com)wrote: > Hello, > > I am new to elasticsearch. I am trying use > [GitHub - eBay/cassandra-river: Cassandra river for Elastic search.](https://github.com/eBay/cassandra-river) plugin to store data to > cassandra. > I dont see the data in cassandra. My search queries are returning from the > elasticsearch. > > Please advise. > > Thanks, > Ashwini > > On Friday, March 29, 2013 2:01:14 PM UTC-7, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > > > Hello, > > > > I have been working on a cassandra river which triggers periodically and > > indexes all data in a cassandra column family. The implementation for now > > spawns 10 threads and processes 10k documents (with 13 columns)/thread. > > The performance initially was very good. It indexed 1M documents in > > 10mins. But after a 1hour, the indexing became very slow and it indexed > > around 8M documents. I am trying to index a total of 50M documents. > > > > I have attached a screenshot of the memory and CPU usage. What I noticed > > was, a lot of merge threads spawned up which reduced the speed considerably: > > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" > > daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > > > So, I believe this has to do with some configuration which I can tweak to > > improve bulk indexing. I am running 1 node with 5 shared with 2GB of > > ES\_HEAP\_SIZE and no replicas for now. > > > > Shay mentioned some tips here: [https://groups.google.com/](https://groups.google.com/) > > forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU in 2011. > > Wanted to know if there are any bulk indexing performance improvements? > > > > I am also using: bulk.execute().addListener() (async) in place of > > bulk.execute().actionGet() (sync) > > > > I am planning to share the cassandra-river as soon its achieves acceptable > > performance. > > > > [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) > > > > [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) > > > > Thanks, > > -Utkarsh > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/M1aJqvAIpZE/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M1aJqvAIpZE/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Thanks, -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 19, 2013, 11:33pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/6 "2013-11-19T23:33:00Z")
I think this river index docs from cassandra to Elasticsearch. Not the opposite. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 19 nov. 2013 à 23:41, Ashwini Lohit [ashwini.2284@gmail.com](mailto:ashwini.2284@gmail.com) a écrit : Hello, I am new to elasticsearch. I am trying use [GitHub - eBay/cassandra-river: Cassandra river for Elastic search.](https://github.com/eBay/cassandra-river) plugin to store data to cassandra. I dont see the data in cassandra. My search queries are returning from the elasticsearch. Please advise. Thanks, Ashwini > On Friday, March 29, 2013 2:01:14 PM UTC-7, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > Hello, > > I have been working on a cassandra river which triggers periodically and indexes all data in a cassandra column family. The implementation for now spawns 10 threads and processes 10k documents (with 13 columns)/thread. > The performance initially was very good. It indexed 1M documents in 10mins. But after a 1hour, the indexing became very slow and it indexed around 8M documents. I am trying to index a total of 50M documents. > > I have attached a screenshot of the memory and CPU usage. What I noticed was, a lot of merge threads spawned up which reduced the speed considerably: > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > So, I believe this has to do with some configuration which I can tweak to improve bulk indexing. I am running 1 node with 5 shared with 2GB of ES\_HEAP\_SIZE and no replicas for now. > > Shay mentioned some tips here: [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU) in 2011. > Wanted to know if there are any bulk indexing performance improvements? > > I am also using: bulk.execute().addListener() (async) in place of bulk.execute().actionGet() (sync) > > I am planning to share the cassandra-river as soon its achieves acceptable performance. > > Thanks, > -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ashwini](https://avatars.discourse-cdn.com/v4/letter/a/b5ac83/32.png) [@ashwini](https://discuss.elastic.co/u/ashwini)\ **Post date:** [November 22, 2013, 7:20pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/7 "2013-11-22T19:20:28Z")
On Friday, March 29, 2013 2:01:14 PM UTC-7, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > Hello, > > I have been working on a cassandra river which triggers periodically and > indexes all data in a cassandra column family. The implementation for now > spawns 10 threads and processes 10k documents (with 13 columns)/thread. > The performance initially was very good. It indexed 1M documents in > 10mins. But after a 1hour, the indexing became very slow and it indexed > around 8M documents. I am trying to index a total of 50M documents. > > I have attached a screenshot of the memory and CPU usage. What I noticed > was, a lot of merge threads spawned up which reduced the speed considerably: > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" > daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > So, I believe this has to do with some configuration which I can tweak to > improve bulk indexing. I am running 1 node with 5 shared with 2GB of > ES\_HEAP\_SIZE and no replicas for now. > > Shay mentioned some tips here: > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeUin) 2011. > Wanted to know if there are any bulk indexing performance improvements? > > I am also using: bulk.execute().addListener() (async) in place of > bulk.execute().actionGet() (sync) > > I am planning to share the cassandra-river as soon its achieves acceptable > performance. > > [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) > > [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) > > Thanks, > -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 22, 2013, 8:58pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/8 "2013-11-22T20:58:59Z")
Moving this to the ML. Did you restart the node? What are the 15 first lines of your node logs? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 22 nov. 2013 à 21:41, Ashwini Lohit [ashwini.2284@gmail.com](mailto:ashwini.2284@gmail.com) a écrit : > Logs: > > [2013-11-22 19:33:48,991][DEBUG][river.cluster] [Gigantus] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state > [2013-11-22 19:33:49,993][DEBUG][river.cluster] [Gigantus] processing [reroute\_rivers\_node\_changed]: execute > [2013-11-22 19:33:50,014][INFO][river.routing] [Gigantus] no river \_meta document found, retrying in 1000 ms > [2013-11-22 19:33:50,014][DEBUG][river.cluster] [Gigantus] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state > [2013-11-22 19:33:51,017][DEBUG][river.cluster] [Gigantus] processing [reroute\_rivers\_node\_changed]: execute > [2013-11-22 19:33:51,018][WARN][river.routing] [Gigantus] no river \_meta document found after 5 attempts > [2013-11-22 19:33:51,018][DEBUG][river.cluster] [Gigantus] processing [reroute\_rivers\_node\_changed]: no change in cluster\_state > [2013-11-22 19:35:09,915][DEBUG][cluster.service] [Gigantus] processing [update-mapping [\_river][test]]: execute > [2013-11-22 19:35:09,916][DEBUG][cluster.metadata] [Gigantus] [\_river] update\_mapping [test] ignoring mapping update task as it's source is equal to ours > [2013-11-22 19:35:09,917][DEBUG][cluster.service] [Gigantus] processing [update-mapping [\_river][test]]: no change in cluster\_state > > This is all the info I see in the logs. Looks like the river is loaded. > > Thanks, > Ashwini > > On Fri, Nov 22, 2013 at 12:32 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote: > > > Anything in logs? > > Is the river loaded? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 22 nov. 2013 à 20:21, Ashwini Lohit [ashwini.2284@gmail.com](mailto:ashwini.2284@gmail.com) a écrit : > > > > > Hello, > > > > > > Thanks for the reply. I might have some misconfiguration in the settings. > > > > > > Version: > > > Elasticsearch : 0.90.5 > > > Cassandra: 1.1.6 > > > Cassandra River - [GitHub - eBay/cassandra-river: Cassandra river for Elastic search.](https://github.com/eBay/cassandra-river) (latest) > > > > > > I did the following steps > > > > > > 1. Created an instance of cassandra > > > 2. Created keyspace test > > > 3. Created column\_family > > > 4. Populated 10 records > > > like this > > > [default@test] SET users['bobbyjo']['full\_name']='Robert Jones'; > > > > > > [default@test] SET users['bobbyjo']['email']='bobjones@gmail.com'; > > > > > > [default@test] SET users['bobbyjo']['state']='TX'; > > > > > > [default@test] SET users['bobbyjo']['gender']='M'; > > > > > > [default@test] SET users['bobbyjo']['birth\_year']='1975'; > > > 5. Installed elasticsearch cassandra river plugin > > > 6.curl -XPUT "http://:9200/\_river/abc/\_meta" -d' > > > { > > > "type" : "cassandra", > > > "cassandra" : { > > > "cluster\_name" : "demo", > > > "keyspace" : "test", > > > "column\_family" : "users", > > > "hosts" : ":9160" > > > }, > > > "index" : { > > > "index" : "abc", > > > "type" : "cassandra" > > > } > > > }' > > > > > > I still dont see the data being indexed in Elasticsearch, here is what I ran to look at it. > > > curl -XGET "http://:9200/abc/cassandra/\_search" -d' > > > { > > > "query": { > > > "match\_all": { > > > > > > ``` > > > } > > > } > > > > > > ``` > > > > > > }' > > > I also tried > > > > > > curl -XGET "http://:9200/users/\_search" -d' > > > { > > > "query": { > > > "match\_all": { > > > > > > ``` > > > } > > > } > > > > > > ``` > > > > > > }' > > > > > > Please advise. > > > > > > Thanks, > > > Ashwini > > > On Tuesday, November 19, 2013 3:33:00 PM UTC-8, David Pilato wrote: > > > > > > > I think this river index docs from cassandra to Elasticsearch. > > > > Not the opposite. > > > > > > > > -- > > > > David 😉 > > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > > > > > Le 19 nov. 2013 à 23:41, Ashwini Lohit [ashwin...@gmail.com](mailto:ashwin...@gmail.com) a écrit : > > > > > > > > Hello, > > > > > > > > I am new to elasticsearch. I am trying use [GitHub - eBay/cassandra-river: Cassandra river for Elastic search.](https://github.com/eBay/cassandra-river) plugin to store data to cassandra. > > > > I dont see the data in cassandra. My search queries are returning from the elasticsearch. > > > > > > > > Please advise. > > > > > > > > Thanks, > > > > Ashwini > > > > > > > > On Friday, March 29, 2013 2:01:14 PM UTC-7, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > > > > > > > > > Hello, > > > > > > > > > > I have been working on a cassandra river which triggers periodically and indexes all data in a cassandra column family. The implementation for now spawns 10 threads and processes 10k documents (with 13 columns)/thread. > > > > > The performance initially was very good. It indexed 1M documents in 10mins. But after a 1hour, the indexing became very slow and it indexed around 8M documents. I am trying to index a total of 50M documents. > > > > > > > > > > I have attached a screenshot of the memory and CPU usage. What I noticed was, a lot of merge threads spawned up which reduced the speed considerably: > > > > > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > > > > > > > > > So, I believe this has to do with some configuration which I can tweak to improve bulk indexing. I am running 1 node with 5 shared with 2GB of ES\_HEAP\_SIZE and no replicas for now. > > > > > > > > > > Shay mentioned some tips here: [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU) in 2011. > > > > > Wanted to know if there are any bulk indexing performance improvements? > > > > > > > > > > I am also using: bulk.execute().addListener() (async) in place of bulk.execute().actionGet() (sync) > > > > > > > > > > I am planning to share the cassandra-river as soon its achieves acceptable performance. > > > > > > > > > > Thanks, > > > > > -Utkarsh > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![AKhan](https://avatars.discourse-cdn.com/v4/letter/a/34f0e0/32.png) [@AKhan](https://discuss.elastic.co/u/AKhan)\ **Post date:** [February 4, 2014, 4:54pm UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/9 "2014-02-04T16:54:30Z")
cassandra-river is not working in my case too and I am getting exceptions on server side. elasticsearch.common.UUID; On Friday, March 29, 2013 10:01:14 PM UTC+1, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > Hello, > > I have been working on a cassandra river which triggers periodically and > indexes all data in a cassandra column family. The implementation for now > spawns 10 threads and processes 10k documents (with 13 columns)/thread. > The performance initially was very good. It indexed 1M documents in > 10mins. But after a 1hour, the indexing became very slow and it indexed > around 8M documents. I am trying to index a total of 50M documents. > > I have attached a screenshot of the memory and CPU usage. What I noticed > was, a lot of merge threads spawned up which reduced the speed considerably: > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" > daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > So, I believe this has to do with some configuration which I can tweak to > improve bulk indexing. I am running 1 node with 5 shared with 2GB of > ES\_HEAP\_SIZE and no replicas for now. > > Shay mentioned some tips here: > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeUin) 2011. > Wanted to know if there are any bulk indexing performance improvements? > > I am also using: bulk.execute().addListener() (async) in place of > bulk.execute().actionGet() (sync) > > I am planning to share the cassandra-river as soon its achieves acceptable > performance. > > [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) > > [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) > > Thanks, > -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1f5550ca-d53e-4513-b691-8992e0504533%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1f5550ca-d53e-4513-b691-8992e0504533%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![utkarsh2012](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/utkarsh2012/32/1187_2.png) [@utkarsh2012](https://discuss.elastic.co/u/utkarsh2012)\ **Post date:** [February 5, 2014, 12:18am UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/10 "2014-02-05T00:18:38Z")
Can you please file a bug ([Issues · eBay/cassandra-river · GitHub](https://github.com/eBay/cassandra-river/issues)) or share the stacktrace? Thanks, -Utkarsh On Tue, Feb 4, 2014 at 8:54 AM, AKhan [ansark6@gmail.com](mailto:ansark6@gmail.com) wrote: > cassandra-river is not working in my case too and I am getting exceptions > on server side. > > elasticsearch.common.UUID; > > On Friday, March 29, 2013 10:01:14 PM UTC+1, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote: > > > Hello, > > > > I have been working on a cassandra river which triggers periodically and > > indexes all data in a cassandra column family. The implementation for now > > spawns 10 threads and processes 10k documents (with 13 columns)/thread. > > The performance initially was very good. It indexed 1M documents in > > 10mins. But after a 1hour, the indexing became very slow and it indexed > > around 8M documents. I am trying to index a total of 50M documents. > > > > I have attached a screenshot of the memory and CPU usage. What I noticed > > was, a lot of merge threads spawned up which reduced the speed considerably: > > "elasticsearch[Doppelganger][[prodinfo][1]: Lucene Merge Thread #329]" > > daemon prio=10 tid=0x2a630000 nid=0x4c28 runnable [0x246bd000] > > > > So, I believe this has to do with some configuration which I can tweak to > > improve bulk indexing. I am running 1 node with 5 shared with 2GB of > > ES\_HEAP\_SIZE and no replicas for now. > > > > Shay mentioned some tips here: [https://groups.google.com/](https://groups.google.com/) > > forum/?fromgroups=#!topic/elasticsearch/APWxRLrMOeU in 2011. > > Wanted to know if there are any bulk indexing performance improvements? > > > > I am also using: bulk.execute().addListener() (async) in place of > > bulk.execute().actionGet() (sync) > > > > I am planning to share the cassandra-river as soon its achieves acceptable > > performance. > > > > [https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk\_ES.png](https://lh5.googleusercontent.com/-G8kxNXFaUmc/UVYAPHN-NFI/AAAAAAAAAgw/rmcLu6P1Urg/s1600/bigdesk_ES.png) > > > > [https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm\_Es.png](https://lh3.googleusercontent.com/-8RokQMwPSW0/UVYAJFwUvrI/AAAAAAAAAgo/GTEdX5MkTwA/s1600/visualvm_Es.png) > > > > Thanks, > > -Utkarsh > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/M1aJqvAIpZE/unsubscribe](https://groups.google.com/d/topic/elasticsearch/M1aJqvAIpZE/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1f5550ca-d53e-4513-b691-8992e0504533%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1f5550ca-d53e-4513-b691-8992e0504533%40googlegroups.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Thanks, -Utkarsh -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CADjjot9uWLyBw%3D7ESYL%3D-sHLJmBLigPx4iON1UjsXNgpWsm18g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CADjjot9uWLyBw%3D7ESYL%3D-sHLJmBLigPx4iON1UjsXNgpWsm18g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:52am UTC](https://discuss.elastic.co/t/bulk-indexing-tips-for-elastic-search-and-cassandra-river/11387/11 "2017-07-06T01:52:36Z")