# Elastic search **URL:** \ **Category:** Elasticsearch\ **Created:** [August 7, 2011, 7:04pm UTC](https://discuss.elastic.co/t/elastic-search/5076 "2011-08-07T19:04:34Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [August 7, 2011, 7:04pm UTC](https://discuss.elastic.co/t/elastic-search/5076/1 "2011-08-07T19:04:34Z")
I am new to elastic search and trying to understand the concept. I am trying to find the information: 1. about how it distributes, replicates data for HA. 2. Where does it store the data? 3. Optimization techniques ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [August 7, 2011, 10:56pm UTC](https://discuss.elastic.co/t/elastic-search/5076/2 "2011-08-07T22:56:45Z")
1. ES shards and replicates indexes. It is what I would call 'statically sharded' - that is you specify up front the number of shards and replicas you want and that's how many there will be. Shards and replicas are then allocated to nodes in your cluster. 2. Up to you: [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) 3. Depends upon your use case. Everyone's data and everyone's indexes will be different. On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > I am new to Elasticsearch and trying to understand the concept. I am > trying to find the information: > > 1. about how it distributes, replicates data for HA. > 2. Where does it store the data? > 3. Optimization techniques ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [August 8, 2011, 5:34am UTC](https://discuss.elastic.co/t/elastic-search/5076/3 "2011-08-08T05:34:03Z")
On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > 1. ES shards and replicates indexes. It is what I would call 'statically > sharded' - that is you specify up front the number of shards and replicas > you want and that's how many there will be. Shards and replicas are then > allocated to nodes in your cluster. Is there a link where I can read how to configure that? Also, does it make it HA for eg: if on enode goes down then it doesn't impact the searching? > 1. Up to you: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) How to decide which one to use? I also see it integrates with CouchDB. When having TBs of data is it ok to keep on the file system? > 1. Depends upon your use case. Everyone's data and everyone's indexes will > be different. Are there any general guidelines that might be applicable to everyone or at least gives litte more thought processing into design this efficiently? > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > I am new to Elasticsearch and trying to understand the concept. I am > > trying to find the information: > > > > 1. about how it distributes, replicates data for HA. > > 2. Where does it store the data? > > 3. Optimization techniques > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Paul\_Loy](https://avatars.discourse-cdn.com/v4/letter/p/ad7895/32.png) [@Paul\_Loy](https://discuss.elastic.co/u/Paul_Loy)\ **Post date:** [August 8, 2011, 11:19am UTC](https://discuss.elastic.co/t/elastic-search/5076/4 "2011-08-08T11:19:47Z")
On Mon, Aug 8, 2011 at 6:34 AM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote: > On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > 1. ES shards and replicates indexes. It is what I would call 'statically > > sharded' - that is you specify up front the number of shards and replicas > > you want and that's how many there will be. Shards and replicas are then > > allocated to nodes in your cluster. > > Is there a link where I can read how to configure that? Also, does it > make it HA for eg: if on enode goes down then it doesn't impact the > searching? Basic configuration will be the index settings where you can set the number of shards and the number of replicas of an index. > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... What's awesome with ES is that you can specify this on a per index basis. So more critical indices can have a higher number of replicas. Regarding HA, the was I understand it (and Shay can probably correct me if I'm wrong), there is a 'master' node for a shard. If that node dies, another node with a replica is voted the 'master'. So searches should not be impacted if a node goes down. Obviously if you had enough nodes for one per shard and a node goes down then one node will now have to do 2 shards of searches and so may be slower. So while you can still run searches, you'll need to think about redundancy in your cluster. > > 1. Up to you: > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > How to decide which one to use? I also see it integrates with CouchDB. > When having TBs of data is it ok to keep on the file system? This will be better answered by one of the guys on this list that also pushes TBs of data. I'm only at the GBs size so I use S3 for a gateway just to be sure. I guess the quick answer is you can scale out to meet your needs! If FS is a bottleneck you can add more nodes!? > > 1. Depends upon your use case. Everyone's data and everyone's indexes > > will > > be different. > > Are there any general guidelines that might be applicable to everyone > or at least gives litte more thought processing into design this > efficiently? Lots, and it really is dependent on your data and how you want to search it. Some tips I've used are to use filters as much as possible, which seems to have given us a very stable, low latency ES cluster. > > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > > > I am new to Elasticsearch and trying to understand the concept. I am > > > trying to find the information: > > > > > > 1. about how it distributes, replicates data for HA. > > > 2. Where does it store the data? > > > 3. Optimization techniques > > > > ## -- > > > > Paul Loy > > [paul@keteracel.com](mailto:paul@keteracel.com) > > [Paul Loy - Amihan Entertainment | LinkedIn](http://uk.linkedin.com/in/paulloy) ## -- Paul Loy [paul@keteracel.com](mailto:paul@keteracel.com) [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [August 10, 2011, 3:51pm UTC](https://discuss.elastic.co/t/elastic-search/5076/5 "2011-08-10T15:51:25Z")
Are there any recommendation as to when to use DB compared to file system? Our use case is simple: 1. We have tons of column name and values in NoSQL column families that we need to have search capabilities on since NoSQL cassandra isn't really very good when you need lots of indexes. These are mostly distinct values. 2. We have xml docs that have attributes that we need to search for. These have low cardinality. On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > 1. ES shards and replicates indexes. It is what I would call 'statically > sharded' - that is you specify up front the number of shards and replicas > you want and that's how many there will be. Shards and replicas are then > allocated to nodes in your cluster. > > 2. Up to you: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > 3. Depends upon your use case. Everyone's data and everyone's indexes will > be different. > > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > I am new to Elasticsearch and trying to understand the concept. I am > > trying to find the information: > > > > 1. about how it distributes, replicates data for HA. > > 2. Where does it store the data? > > 3. Optimization techniques > > ## -- > > Paul Loy > [paul@keteracel.com](mailto:paul@keteracel.com) > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 13, 2011, 4:45pm UTC](https://discuss.elastic.co/t/elastic-search/5076/6 "2011-08-13T16:45:16Z")
What kind of recommendations are you after? Not sure I understand the question properly to answer it... On Wed, Aug 10, 2011 at 6:51 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote: > Are there any recommendation as to when to use DB compared to file system? > > Our use case is simple: > > 1. We have tons of column name and values in NoSQL column families > that we need to have search capabilities on since NoSQL cassandra > isn't really very good when you need lots of indexes. These are mostly > distinct values. > 2. We have xml docs that have attributes that we need to search for. > These have low cardinality. > > On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > 1. ES shards and replicates indexes. It is what I would call 'statically > > sharded' - that is you specify up front the number of shards and replicas > > you want and that's how many there will be. Shards and replicas are then > > allocated to nodes in your cluster. > > > > 2. Up to you: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > 3. Depends upon your use case. Everyone's data and everyone's indexes > > will > > be different. > > > > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > > > I am new to Elasticsearch and trying to understand the concept. I am > > > trying to find the information: > > > > > > 1. about how it distributes, replicates data for HA. > > > 2. Where does it store the data? > > > 3. Optimization techniques > > > > ## -- > > > > Paul Loy > > [paul@keteracel.com](mailto:paul@keteracel.com) > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [August 13, 2011, 5:25pm UTC](https://discuss.elastic.co/t/elastic-search/5076/7 "2011-08-13T17:25:55Z")
On Sat, Aug 13, 2011 at 9:45 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > What kind of recommendations are you after? Not sure I understand the > question properly to answer it... How to decide to use File system or CouchDB? What would be the reason people would chose one over other? Is it just because you can see data in some form directly in the DB? > On Wed, Aug 10, 2011 at 6:51 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) > wrote: > > > Are there any recommendation as to when to use DB compared to file system? > > > > Our use case is simple: > > > > 1. We have tons of column name and values in NoSQL column families > > that we need to have search capabilities on since NoSQL cassandra > > isn't really very good when you need lots of indexes. These are mostly > > distinct values. > > 2. We have xml docs that have attributes that we need to search for. > > These have low cardinality. > > > > On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > 1. ES shards and replicates indexes. It is what I would call 'statically > > > sharded' - that is you specify up front the number of shards and > > > replicas > > > you want and that's how many there will be. Shards and replicas are then > > > allocated to nodes in your cluster. > > > > > > 2. Up to you: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > 3. Depends upon your use case. Everyone's data and everyone's indexes > > > will > > > be different. > > > > > > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > > > > > I am new to Elasticsearch and trying to understand the concept. I am > > > > trying to find the information: > > > > > > > > 1. about how it distributes, replicates data for HA. > > > > 2. Where does it store the data? > > > > 3. Optimization techniques > > > > > > ## -- > > > > > > Paul Loy > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 13, 2011, 7:43pm UTC](https://discuss.elastic.co/t/elastic-search/5076/8 "2011-08-13T19:43:44Z")
Still not understanding... . Use a file system or use couchdb? How does relate to elasticsearch? If not, I can still try and help :), but need more info, you want to store blobs on the file system? On Sat, Aug 13, 2011 at 8:25 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com)wrote: > On Sat, Aug 13, 2011 at 9:45 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > What kind of recommendations are you after? Not sure I understand the > > question properly to answer it... > > How to decide to use File system or CouchDB? What would be the reason > people would chose one over other? Is it just because you can see data > in some form directly in the DB? > > > On Wed, Aug 10, 2011 at 6:51 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) > > wrote: > > > > > Are there any recommendation as to when to use DB compared to file > > > system? > > > > > > Our use case is simple: > > > > > > 1. We have tons of column name and values in NoSQL column families > > > that we need to have search capabilities on since NoSQL cassandra > > > isn't really very good when you need lots of indexes. These are mostly > > > distinct values. > > > 2. We have xml docs that have attributes that we need to search for. > > > These have low cardinality. > > > > > > On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > > > 1. ES shards and replicates indexes. It is what I would call > > > > 'statically > > > > sharded' - that is you specify up front the number of shards and > > > > replicas > > > > you want and that's how many there will be. Shards and replicas are > > > > then > > > > allocated to nodes in your cluster. > > > > > > > > 2. Up to you: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > > > 3. Depends upon your use case. Everyone's data and everyone's indexes > > > > will > > > > be different. > > > > > > > > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > > > > > > > I am new to Elasticsearch and trying to understand the concept. I am > > > > > trying to find the information: > > > > > > > > > > 1. about how it distributes, replicates data for HA. > > > > > 2. Where does it store the data? > > > > > 3. Optimization techniques > > > > > > > > ## -- > > > > > > > > Paul Loy > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [August 14, 2011, 4:38am UTC](https://discuss.elastic.co/t/elastic-search/5076/9 "2011-08-14T04:38:20Z")
On Sat, Aug 13, 2011 at 12:43 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Still not understanding... . Use a file system or use couchdb? How does > relate to elasticsearch? If not, I can still try and help :), but need more > info, you want to store blobs on the file system? From what I understand indexes are stored somewhere on the disk. And from the link [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) it looks like you have various options. So I am trying to understand if it should be stored on the file system or some DB like couchDB? Doesn't elasticsearch store indexed data somewhere? > On Sat, Aug 13, 2011 at 8:25 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) > wrote: > > > On Sat, Aug 13, 2011 at 9:45 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > What kind of recommendations are you after? Not sure I understand the > > > question properly to answer it... > > > > How to decide to use File system or CouchDB? What would be the reason > > people would chose one over other? Is it just because you can see data > > in some form directly in the DB? > > > > > On Wed, Aug 10, 2011 at 6:51 PM, Mohit Anchlia [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) > > > wrote: > > > > > > > Are there any recommendation as to when to use DB compared to file > > > > system? > > > > > > > > Our use case is simple: > > > > > > > > 1. We have tons of column name and values in NoSQL column families > > > > that we need to have search capabilities on since NoSQL cassandra > > > > isn't really very good when you need lots of indexes. These are mostly > > > > distinct values. > > > > 2. We have xml docs that have attributes that we need to search for. > > > > These have low cardinality. > > > > > > > > On Sun, Aug 7, 2011 at 3:56 PM, Paul Loy [keteracel@gmail.com](mailto:keteracel@gmail.com) wrote: > > > > > > > > > 1. ES shards and replicates indexes. It is what I would call > > > > > 'statically > > > > > sharded' - that is you specify up front the number of shards and > > > > > replicas > > > > > you want and that's how many there will be. Shards and replicas are > > > > > then > > > > > allocated to nodes in your cluster. > > > > > > > > > > 2. Up to you: > > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html) > > > > > > > > > > 3. Depends upon your use case. Everyone's data and everyone's indexes > > > > > will > > > > > be different. > > > > > > > > > > On Sun, Aug 7, 2011 at 8:04 PM, Mo [mohitanchlia@gmail.com](mailto:mohitanchlia@gmail.com) wrote: > > > > > > > > > > > I am new to Elasticsearch and trying to understand the concept. I > > > > > > am > > > > > > trying to find the information: > > > > > > > > > > > > 1. about how it distributes, replicates data for HA. > > > > > > 2. Where does it store the data? > > > > > > 3. Optimization techniques > > > > > > > > > > ## -- > > > > > > > > > > Paul Loy > > > > > [paul@keteracel.com](mailto:paul@keteracel.com) > > > > > [http://uk.linkedin.com/in/paulloy](http://uk.linkedin.com/in/paulloy) ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [August 14, 2011, 1:00pm UTC](https://discuss.elastic.co/t/elastic-search/5076/10 "2011-08-14T13:00:55Z")
Hi Mo, There seems to be a disconnect with your questions and some fundamental understanding of how ES (and Lucene) works. I think you need to read the website a bit more, especially take a look at the video: [http://www.elasticsearch.org/videos/2011/08/09/road-to-a-distributed-searchengine-berlinbuzzwords.html](http://www.elasticsearch.org/videos/2011/08/09/road-to-a-distributed-searchengine-berlinbuzzwords.html) Index storage is under the control of Lucene, and the store page[http://www.elasticsearch.org/guide/reference/index-modules/store.html](http://www.elasticsearch.org/guide/reference/index-modules/store.html)you link to describes your options with simplefs, niofs being file-based, memory being memory-based, and mmapfs being a hybrid of the two. I'm not sure where you got the idea that indexes can also be stored in a DB like CouchDB. There is the concept of a River which is a bridge between CouchDB (and others) and ES. A River will receive push changes or periodically will pull changes from a source (like CouchDB, not sure if CouchDB River pushes or pulls) and index the data it receives. This is a technique that can be used to put things for searching into ES without the developer having to specifically index documents into ES. It has nothing to do with how data is stored in ES. -- jim ---
**Author:** ![Mohit\_Anchlia](https://avatars.discourse-cdn.com/v4/letter/m/848f3c/32.png) [@Mohit\_Anchlia](https://discuss.elastic.co/u/Mohit_Anchlia)\ **Post date:** [August 14, 2011, 10:41pm UTC](https://discuss.elastic.co/t/elastic-search/5076/11 "2011-08-14T22:41:34Z")
On Sun, Aug 14, 2011 at 6:00 AM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote: > Hi Mo, > There seems to be a disconnect with your questions and some fundamental > understanding of how ES (and Lucene) works. I think you need to read the > website a bit more, especially take a look at the video: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2011/08/09/road-to-a-distributed-searchengine-berlinbuzzwords.html) > > Index storage is under the control of Lucene, and the store page you link to > describes your options with simplefs, niofs being file-based, memory being > memory-based, and mmapfs being a hybrid of the two. I'm not sure where you > got the idea that indexes can also be stored in a DB like CouchDB. > There is the concept of a River which is a bridge between CouchDB (and > others) and ES. A River will receive push changes or periodically will pull > changes from a source (like CouchDB, not sure if CouchDB River pushes or > pulls) and index the data it receives. This is a technique that can be used > to put things for searching into ES without the developer having to > specifically index documents into ES. It has nothing to do with how data is > stored in ES. Thanks for clarifying. I will go through that presentation. > -- jim ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:57am UTC](https://discuss.elastic.co/t/elastic-search/5076/12 "2017-07-06T03:57:02Z")