# I've done a Cassandra gateway implementation, and have some questions **URL:** \ **Category:** Elasticsearch\ **Created:** [December 2, 2010, 8:14pm UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620 "2010-12-02T20:14:22Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![tommay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tommay/32/3295_2.png) [@tommay](https://discuss.elastic.co/u/tommay)\ **Post date:** [December 2, 2010, 8:14pm UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620/1 "2010-12-02T20:14:22Z")
I've done a Cassandra gateway implementation as a plugin at [https://github.com/gistinc/elasticsearch/tree/cassandra](https://github.com/gistinc/elasticsearch/tree/cassandra). It's built with ElasticSearch 0.13.0. I'm using Cassandra 0.6.8. I'm wondering how Cassandra's eventual consistency model interacts with how ElasticSearch uses its gateway. I admit I have no idea how ElasticSearch uses its gateway, especially from multiple nodes in the cluster. It seems safest to use Cassandra's QUORUM consistency level (which requires at least three Cassandra nodes to tolerate one failing), but it's not clear whether multiple ElasticSearch nodes might be writing the same blob concurrently, and what they expect to happen if they do. Can somebody enlighten me about that? The Cassandra gateway is not much beyond a proof-of-concept but it does work and we're currently testing with it at Gist [http://www.gist.com/](http://www.gist.com/). Its biggest limitation is that it only talks to a single Cassandra server. It's using a bare thrift interface; I'll be looking into using an existing higher-level Java interface that supports failover and such. I'm not sure how well it will handle very large blobs. Handling large blobs wasn't a goal. We intend to use this with per-user indexes that won't get very large. We're also using it with the code on the gistlru branch of that same git repo, written by my co-worker Matt, which allows us to use index.store.type = memory and keep only indexes for active users in memory, the rest being persisted to the gateway. We hope. Tom ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 3, 2010, 12:51am UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620/2 "2010-12-03T00:51:08Z")
On Thu, Dec 2, 2010 at 10:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote: > I've done a Cassandra gateway implementation as a plugin at > [GitHub - gistinc/elasticsearch at cassandra](https://github.com/gistinc/elasticsearch/tree/cassandra). It's built with > Elasticsearch 0.13.0. I'm using Cassandra 0.6.8. Great job!, once its settled and done, we can maybe think about getting it into elasticsearch itself, though my Cassandra knowledge is quite lacking 🙂 > I'm wondering how Cassandra's eventual consistency model interacts with how > Elasticsearch uses its gateway. I admit I have no idea how Elasticsearch > uses its gateway, especially from multiple nodes in the cluster. It seems > safest to use Cassandra's QUORUM consistency level (which requires at least > three Cassandra nodes to tolerate one failing), but it's not clear whether > multiple Elasticsearch nodes might be writing the same blob concurrently, > and what they expect to happen if they do. Can somebody enlighten me about > that? Only the primary shard in each shard replication group writes to the blob store. So there is only a single writer per shard path. I think QUORUM is a good value, as if it will fail, it will simply retry next time. Reading from the blob store only happens when a primary shard is allocated and it needs to recover its data. > The Cassandra gateway is not much beyond a proof-of-concept but it does > work and we're currently testing with it at Gist [http://www.gist.com/](http://www.gist.com/). > Its biggest limitation is that it only talks to a single Cassandra server. > It's using a bare thrift interface; I'll be looking into using an existing > higher-level Java interface that supports failover and such. Sounds great!. > I'm not sure how well it will handle very large blobs. Handling large > blobs wasn't a goal. We intend to use this with per-user indexes that won't > get very large. We're also using it with the code on the gistlru branch of > that same git repo, written by my co-worker Matt, which allows us to use > index.store.type = memory and keep only indexes for active users in memory, > the rest being persisted to the gateway. We hope. There is already automatic support for that. If you set gateway.blobstore.chunk\_size then files will be automatically broken down into several blobs. This is how it is done with the s3 gateway, and it actually automatically set the default chunk size to be 100mb. > Tom ---
**Author:** ![tommay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tommay/32/3295_2.png) [@tommay](https://discuss.elastic.co/u/tommay)\ **Post date:** [December 3, 2010, 1:05am UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620/3 "2010-12-03T01:05:27Z")
On Thu, Dec 2, 2010 at 4:51 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > On Thu, Dec 2, 2010 at 10:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote: > > > I've done a Cassandra gateway implementation as a plugin at > > [GitHub - gistinc/elasticsearch at cassandra](https://github.com/gistinc/elasticsearch/tree/cassandra). It's built with > > Elasticsearch 0.13.0. I'm using Cassandra 0.6.8. > > Great job!, once its settled and done, we can maybe think about getting it > into elasticsearch itself, though my Cassandra knowledge is quite lacking 🙂 Thanks. I'm all for it being included in Elasticsearch at some point. And I've just realized I could factor things more cleanly so I'm working on that 🙂 > > I'm wondering how Cassandra's eventual consistency model interacts with > > how Elasticsearch uses its gateway. I admit I have no idea how > > Elasticsearch uses its gateway, especially from multiple nodes in the > > cluster. It seems safest to use Cassandra's QUORUM consistency level (which > > requires at least three Cassandra nodes to tolerate one failing), but it's > > not clear whether multiple Elasticsearch nodes might be writing the same > > blob concurrently, and what they expect to happen if they do. Can somebody > > enlighten me about that? > > Only the primary shard in each shard replication group writes to the blob > store. So there is only a single writer per shard path. I think QUORUM is a > good value, as if it will fail, it will simply retry next time. Reading from > the blob store only happens when a primary shard is allocated and it needs > to recover its data. Good news, thanks. > > The Cassandra gateway is not much beyond a proof-of-concept but it does > > work and we're currently testing with it at Gist [http://www.gist.com/](http://www.gist.com/). > > Its biggest limitation is that it only talks to a single Cassandra server. > > It's using a bare thrift interface; I'll be looking into using an existing > > higher-level Java interface that supports failover and such. > > Sounds great!. > > > I'm not sure how well it will handle very large blobs. Handling large > > blobs wasn't a goal. We intend to use this with per-user indexes that won't > > get very large. We're also using it with the code on the gistlru branch of > > that same git repo, written by my co-worker Matt, which allows us to use > > index.store.type = memory and keep only indexes for active users in memory, > > the rest being persisted to the gateway. We hope. > > There is already automatic support for that. If you > set gateway.blobstore.chunk\_size then files will be automatically broken > down into several blobs. This is how it is done with the s3 gateway, and it > actually automatically set the default chunk size to be 100mb. Ok, I saw that, but I pretty well ignored it for this go, since I could. Tom > > Tom ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 3, 2010, 6:47pm UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620/4 "2010-12-03T18:47:30Z")
I think that in any case, it would make sense to initialize the chunk\_size to a specific value when working with Cassandra, something like 10-50mb. This will increase the snapshotting process as those are done in parallel. Also, it will make cassandra gateway good for any type of usage "out of the box". On Friday, December 3, 2010 at 3:05 AM, Tom May wrote: > On Thu, Dec 2, 2010 at 4:51 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > On Thu, Dec 2, 2010 at 10:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote: > > > > > I've done a Cassandra gateway implementation as a plugin at [GitHub - gistinc/elasticsearch at cassandra](https://github.com/gistinc/elasticsearch/tree/cassandra). It's built with Elasticsearch 0.13.0. I'm using Cassandra 0.6.8. > > > > Great job!, once its settled and done, we can maybe think about getting it into elasticsearch itself, though my Cassandra knowledge is quite lacking 🙂 > > Thanks. I'm all for it being included in Elasticsearch at some point. > > And I've just realized I could factor things more cleanly so I'm working on that 🙂 > > > > I'm wondering how Cassandra's eventual consistency model interacts with how Elasticsearch uses its gateway. I admit I have no idea how Elasticsearch uses its gateway, especially from multiple nodes in the cluster. It seems safest to use Cassandra's QUORUM consistency level (which requires at least three Cassandra nodes to tolerate one failing), but it's not clear whether multiple Elasticsearch nodes might be writing the same blob concurrently, and what they expect to happen if they do. Can somebody enlighten me about that? > > > > Only the primary shard in each shard replication group writes to the blob store. So there is only a single writer per shard path. I think QUORUM is a good value, as if it will fail, it will simply retry next time. Reading from the blob store only happens when a primary shard is allocated and it needs to recover its data. > > Good news, thanks. > > > > The Cassandra gateway is not much beyond a proof-of-concept but it does work and we're currently testing with it at Gist. Its biggest limitation is that it only talks to a single Cassandra server. It's using a bare thrift interface; I'll be looking into using an existing higher-level Java interface that supports failover and such. > > > > Sounds great!. > > > > > I'm not sure how well it will handle very large blobs. Handling large blobs wasn't a goal. We intend to use this with per-user indexes that won't get very large. We're also using it with the code on the gistlru branch of that same git repo, written by my co-worker Matt, which allows us to use index.store.type = memory and keep only indexes for active users in memory, the rest being persisted to the gateway. We hope. > > > > There is already automatic support for that. If you set gateway.blobstore.chunk\_size then files will be automatically broken down into several blobs. This is how it is done with the s3 gateway, and it actually automatically set the default chunk size to be 100mb. > > Ok, I saw that, but I pretty well ignored it for this go, since I could. > > Tom > > > > Tom ---
**Author:** ![tommay](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tommay/32/3295_2.png) [@tommay](https://discuss.elastic.co/u/tommay)\ **Post date:** [December 3, 2010, 7:28pm UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620/5 "2010-12-03T19:28:15Z")
I'll look into it. Tom On Fri, Dec 3, 2010 at 10:47 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > I think that in any case, it would make sense to initialize the > chunk\_size to a specific value when working with Cassandra, something like > 10-50mb. This will increase the snapshotting process as those are done in > parallel. Also, it will make cassandra gateway good for any type of usage > "out of the box". > > On Friday, December 3, 2010 at 3:05 AM, Tom May wrote: > > On Thu, Dec 2, 2010 at 4:51 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > On Thu, Dec 2, 2010 at 10:14 PM, Tom May [tom@tommay.net](mailto:tom@tommay.net) wrote: > > I've done a Cassandra gateway implementation as a plugin at > [GitHub - gistinc/elasticsearch at cassandra](https://github.com/gistinc/elasticsearch/tree/cassandra). It's built with > Elasticsearch 0.13.0. I'm using Cassandra 0.6.8. > > Great job!, once its settled and done, we can maybe think about getting it > into elasticsearch itself, though my Cassandra knowledge is quite lacking 🙂 > > Thanks. I'm all for it being included in Elasticsearch at some point. > > And I've just realized I could factor things more cleanly so I'm working on > that 🙂 > > I'm wondering how Cassandra's eventual consistency model interacts with how > Elasticsearch uses its gateway. I admit I have no idea how Elasticsearch > uses its gateway, especially from multiple nodes in the cluster. It seems > safest to use Cassandra's QUORUM consistency level (which requires at least > three Cassandra nodes to tolerate one failing), but it's not clear whether > multiple Elasticsearch nodes might be writing the same blob concurrently, > and what they expect to happen if they do. Can somebody enlighten me about > that? > > Only the primary shard in each shard replication group writes to the blob > store. So there is only a single writer per shard path. I think QUORUM is a > good value, as if it will fail, it will simply retry next time. Reading from > the blob store only happens when a primary shard is allocated and it needs > to recover its data. > > Good news, thanks. > > > The Cassandra gateway is not much beyond a proof-of-concept but it does > > work and we're currently testing with it at Gist [http://www.gist.com/](http://www.gist.com/). > > Its biggest limitation is that it only talks to a single Cassandra server. > > It's using a bare thrift interface; I'll be looking into using an existing > > higher-level Java interface that supports failover and such. > > > > Sounds great!. > > > > I'm not sure how well it will handle very large blobs. Handling large > > blobs wasn't a goal. We intend to use this with per-user indexes that won't > > get very large. We're also using it with the code on the gistlru branch of > > that same git repo, written by my co-worker Matt, which allows us to use > > index.store.type = memory and keep only indexes for active users in memory, > > the rest being persisted to the gateway. We hope. > > > > There is already automatic support for that. If you > > set gateway.blobstore.chunk\_size then files will be automatically broken > > down into several blobs. This is how it is done with the s3 gateway, and it > > actually automatically set the default chunk size to be 100mb. > > Ok, I saw that, but I pretty well ignored it for this go, since I could. > > Tom > > Tom ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/ive-done-a-cassandra-gateway-implementation-and-have-some-questions/3620/6 "2017-07-06T04:15:46Z")
Closed on July 6, 2017, 4:15am UTC