# Please, tell about the success story about ES usage on production **URL:** \ **Category:** Elasticsearch\ **Created:** [November 30, 2011, 10:33am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022 "2011-11-30T10:33:46Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![vsiryi](https://avatars.discourse-cdn.com/v4/letter/v/7ab992/32.png) [@vsiryi](https://discuss.elastic.co/u/vsiryi)\ **Post date:** [November 30, 2011, 10:33am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/1 "2011-11-30T10:33:46Z")
I want to convince my customer to use the ES. To do this I need success stories about ES usage on production. I tried to find similar information at the ElasticSearch official site and in Google but not found. I'll be very happy if you write how many documents you are indexing and how many approximate number of request is served daily. Some links to production application will be very helpful. Thanks! Best regards, Vitalii Siryi ---
**Author:** ![Alexander\_Reelsen](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Alexander\_Reelsen](https://discuss.elastic.co/u/Alexander_Reelsen)\ **Post date:** [November 30, 2011, 11:33am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/2 "2011-11-30T11:33:54Z")
Hi On Nov 30, 11:33 am, Vitalii Siryi [vitalii.si...@gmail.com](mailto:vitalii.si...@gmail.com) wrote: > I'll be very happy if you write how many documents you are indexing > and how many approximate number of request is served daily. Some links > to production application will be very helpful. > We are using it as a product search engine, you can see it live at > [http://www.lusini.de](http://www.lusini.de) Document count is quite low, more than 200k documents, and currently on one node. We never had performance issues. We are doing lots of facetting queries with about ten facets and of course some filters - queries get a bit slower but are still fast enough for us. In order to get the product data into elasticsearch we have implemented a river, which pulls every n seconds for updates and is streaming JSON data, so we do not have to wait until we got all the data, when downloading big data. We have several thousand updates a day (more likely ten thousand). Before elasticsearch we had a lightning fast but unmaintenable self written solution based on bobo and zoie, which we switched in order to have a more simpler solution, which is understood well by all developers. Of course we do not expose elasticsearch directly to browsers, we have another component in between, which also can do stuff like redirecting certain search terms to landing pages. To be honest, I cannot tell, how much requests are coming in per day, but I guess it is somewhat below 100k. Hope this helps. In case of questions, feel free to ask. --Alexander ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [November 30, 2011, 3:58pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/3 "2011-11-30T15:58:10Z")
[http://www.penpalkidsclub.com/](http://www.penpalkidsclub.com/) Written using ES as the only form of persistence. Went live 7/2011. ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [November 30, 2011, 4:29pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/4 "2011-11-30T16:29:51Z")
James, Was curious what settings/features you consider most important to configure/use when using ES without a secondary persistence mechanism. Perhaps this should be a separate thread - but I'm very curious what your experiences are here. Thanks, --Mike On Wed, Nov 30, 2011 at 10:58 AM, James Cook [jcook@pykl.com](mailto:jcook@pykl.com) wrote: > [http://www.penpalkidsclub.com/](http://www.penpalkidsclub.com/) > > Written using ES as the only form of persistence. Went live 7/2011. ---
**Author:** ![Andy\_2](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Andy\_2](https://discuss.elastic.co/u/Andy_2)\ **Post date:** [December 1, 2011, 1:24am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/5 "2011-12-01T01:24:44Z")
Alexander, What made bobo and zoie unmaintenable? How is elasticsearch more maintenable? You said the bobo/zoie solution "lightning fast." Was it significantly faster than elasticsearch? Thanks. On Nov 30, 6:33 am, Alexander Reelsen [alexander.reel...@googlemail.com](mailto:alexander.reel...@googlemail.com) wrote: > Hi > > On Nov 30, 11:33 am, Vitalii Siryi [vitalii.si...@gmail.com](mailto:vitalii.si...@gmail.com) wrote:\> I'll be very happy if you write how many documents you are indexing > > > and how many approximate number of request is served daily. Some links > > to production application will be very helpful. > > We are using it as a product search engine, you can see it live athttp://www.lusini.de > > Document count is quite low, more than 200k documents, and currently > on one node. We never had performance issues. We are doing lots of > facetting queries with about ten facets and of course some filters - > queries get a bit slower but are still fast enough for us. > > In order to get the product data into elasticsearch we have > implemented a river, which pulls every n seconds for updates and is > streaming JSON data, so we do not have to wait until we got all the > data, when downloading big data. We have several thousand updates a > day (more likely ten thousand). > > Before elasticsearch we had a lightning fast but unmaintenable self > written solution based on bobo and zoie, which we switched in order to > have a more simpler solution, which is understood well by all > developers. > > Of course we do not expose elasticsearch directly to browsers, we have > another component in between, which also can do stuff like redirecting > certain search terms to landing pages. > To be honest, I cannot tell, how much requests are coming in per day, > but I guess it is somewhat below 100k. > > Hope this helps. In case of questions, feel free to ask. > > --Alexander ---
**Author:** ![Alexander\_Reelsen](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Alexander\_Reelsen](https://discuss.elastic.co/u/Alexander_Reelsen)\ **Post date:** [December 1, 2011, 7:58am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/6 "2011-12-01T07:58:40Z")
Hi On Dec 1, 2:24 am, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote: > What made bobo and zoie unmaintenable? How is elasticsearch more > maintenable? > The bobo/zoie implementation was developed by some JEE keen dev, which > meant it had tons of layers and a bad implementation in terms of > accessibility via HTTP - one servlet, one API call where everything > from facetted queries up to suggest was done by appending gazillions > of parameters. I do not want to flame or disregard zoie or bobo here, they are good tools, it was really our implementation which made us switch to ES. 🙂 The good part for us is, that we do not have to care that much about the product - we only hacked a river on top of it. Much less code to care about for us. Makes it more maintainable after all and every developer in the team understands our search solution without digging into lucene/bobo/zoie internals. > You said the bobo/zoie solution "lightning fast." Was it significantly > faster than elasticsearch? > Facetting is real fast with lots of data when using bobo. However as > we do not have that much data in one index, we dont care. We are more > than happy with ES speed. --Alexander ---
**Author:** ![Andy\_2](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Andy\_2](https://discuss.elastic.co/u/Andy_2)\ **Post date:** [December 1, 2011, 8:51am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/7 "2011-12-01T08:51:32Z")
I see. Did you look at Sensei? It's a search engine built using bobo and zoie. Just wondered if Sensei is easier to use. On Dec 1, 2:58 am, Alexander Reelsen [alexander.reel...@googlemail.com](mailto:alexander.reel...@googlemail.com) wrote: > Hi > > On Dec 1, 2:24 am, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote:\> What made bobo and zoie unmaintenable? How is elasticsearch more > > > maintenable? > > The bobo/zoie implementation was developed by some JEE keen dev, which > meant it had tons of layers and a bad implementation in terms of > accessibility via HTTP - one servlet, one API call where everything > from facetted queries up to suggest was done by appending gazillions > of parameters. > > I do not want to flame or disregard zoie or bobo here, they are good > tools, it was really our implementation which made us switch to > ES. 🙂 > > The good part for us is, that we do not have to care that much about > the product - we only hacked a river on top of it. Much less code to > care about for us. Makes it more maintainable after all and every > developer in the team understands our search solution without digging > into lucene/bobo/zoie internals. > > > You said the bobo/zoie solution "lightning fast." Was it significantly > > faster than elasticsearch? > > Facetting is real fast with lots of data when using bobo. However as > we do not have that much data in one index, we dont care. We are more > than happy with ES speed. > > --Alexander ---
**Author:** ![Alexander\_Reelsen](https://avatars.discourse-cdn.com/v4/letter/a/85e7bf/32.png) [@Alexander\_Reelsen](https://discuss.elastic.co/u/Alexander_Reelsen)\ **Post date:** [December 1, 2011, 9:02am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/8 "2011-12-01T09:02:05Z")
Hey On Dec 1, 9:51 am, Andy [selforgani...@gmail.com](mailto:selforgani...@gmail.com) wrote: > I see. > > Did you look at Sensei? It's a search engine built using bobo and > zoie. Just wondered if Sensei is easier to use. > I didnt know about it, when we started investigating elasticsearch. Also, it looks more complex as you have to do more manual tasks to get it up and running (i.e. zookeeper). Most fatal, I read the term "schema" several times in the documentation, did not like that 🙂 --Alexander ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [December 2, 2011, 5:43am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/9 "2011-12-02T05:43:16Z")
Hi Michael, My biggest worries are: - Backup/Restore - Split Brain (really, this is my number one concern. Very destructive and almost no way to recover.) Take a look at these threads: - [http://goo.gl/xIXkj](http://goo.gl/xIXkj) - [http://goo.gl/Z7DKO](http://goo.gl/Z7DKO) -- jim ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [December 2, 2011, 4:08pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/10 "2011-12-02T16:08:49Z")
James, I'm with you on the Backup/Restore. Shay has indicated that it's a high priority. We need it to satisfy enterprisy type customers that always want a stable offsite backup. It would also be a great way to manage pushing/pulling time based indexes from a cluster. We're likely to have an index/day and would like to roll them off the back-end after N days. When a user wants to see data past the N day threshold, it would be nice to simply request the daily file from the tape backup system and import it back into the system. We can accomplish the same thing with exports of the \_source field or even of the original document (XML in our case and we will likely backup both) but having indexes at the ready would be very slick. Not sure I understand the split brain issue but I'll doing some reading up. --Mike On Fri, Dec 2, 2011 at 12:43 AM, James Cook [jcook@pykl.com](mailto:jcook@pykl.com) wrote: > Hi Michael, > > My biggest worries are: > > - Backup/Restore > - Split Brain (really, this is my number one concern. Very destructive > and almost no way to recover.) > > Take a look at these threads: > > - [http://goo.gl/xIXkj](http://goo.gl/xIXkj) > - [Redirecting to Google Groups](http://goo.gl/Z7DKO) > > -- jim ---
**Author:** ![drewr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/drewr/32/7803_2.png) [@drewr](https://discuss.elastic.co/u/drewr)\ **Post date:** [December 2, 2011, 6:42pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/11 "2011-12-02T18:42:32Z")
James Cook wrote: > - Split Brain (really, this is my number one concern. Very destructive > and almost no way to recover.) I suggest you try ZooKeeper discovery. It should make split-brain difficult to encounter. > > > This is an initial implementation of a ZooKeeper-based discovery plugin. > > Usage:… > \- Download ZooKeeper 3.3.3 from http://zookeeper.apache.org/releases.html > \- Unzip the ZooKeeper archive into a directory, rename conf/zoo-sample.cnf into zoo.cnf and modify dataDir= line to point to a directory on your machine. Start ZooKeeper by running bin/zkServer.sh start. > \- Install ZooKeeper plugin to ES > \- Assuming that you are running ZooKeeper on the port 2181 (default), add the following lines to config/elasticsearch.yml file > > \`\`\` > zookeeper: > enabled: true > host: localhost:2181 > discovery: > type: zoo\_keeper > \`\`\` > \- Start ES -Drew ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [December 2, 2011, 8:52pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/12 "2011-12-02T20:52:15Z")
On 2 Dez., 17:08, Michael Sick [michael.s...@serenesoftware.com](mailto:michael.s...@serenesoftware.com) wrote: > James, > > I'm with you on the Backup/Restore. Shay has indicated that it's a high > priority. We need it to satisfy enterprisy type customers that always want > a stable offsite backup. It would also be a great way to manage > pushing/pulling time based indexes from a cluster. We're likely to have an > index/day and would like to roll them off the back-end after N days. Rolling off means in your case delete from disc or avoid searching on them? Here is some code to do rolling indices: > > > Here is some code where a rolling index pattern is implemented. Imagine you have… a logical index named 'tweets', now you want to create every day a new index to keep the indices small (Its a better scalable 'sharding', but only if you have time dependent data). Now, in the proposed code you will have to call rollIndex(maximumIndices) once a day. > > Then the new indices are all 'tagged' with a 'tweets\_roll' alias (for later retrieval -\> imrovable?), there is a group of indices for searching (tweets\_search) and feeding (tweets\_feed). Per default it creates a search alias on all indices and a feed alias only for the very latest. It separates the search and the roll alias as it could be the case that one wants to keep some older indices but do not want to search on it. > > What do you think? Its rather simple but it works - a simple test below the code. > > \`\`\` > private static final String simpleDateString = "yyyy-MM-dd-HH-mm-ss"; > public String rollIndex(int maxRollIndices) { > return rollIndex(getIndexName(), maxRollIndices, maxRollIndices); > } > > public String rollIndex(String indexName, int maxRollIndices, int maxSearchIndices) { > String rollAlias = indexName + "\_roll"; > SimpleDateFormat formatter = new SimpleDateFormat(simpleDateString); > > if (maxRollIndices \< 1 || maxSearchIndices \< 1) > throw new RuntimeException("remaining indices, search indices and feeding indices must be at least 1"); > > // get old aliases > Map\ allRollingAliases = getAliases(rollAlias); > > // always create new index and append aliases > String searchAlias = getSearchIndexName(); > String feedAlias = getFeedIndexName(); > String newIndexName = indexName + "\_" + formatter.format(new Date()); > > createIndex(newIndexName); > addAlias(newIndexName, searchAlias); > addAlias(newIndexName, rollAlias); > String oldFeedIndexName = null; > > if (allRollingAliases.isEmpty()) { > // do nothing for now > } else { > TreeMap\ sortedIndices = new TreeMap\(reverseSorter); > String\[\] concreteIndices = getConcreteIndices(allRollingAliases.keySet()); > //logger.info("aliases:" + allRollingAliases + ", indices:" + Arrays.toString(concreteIndices)); > for (String index : concreteIndices) { > int pos = index.indexOf("\_"); > if (pos \< 0) > throw new IllegalStateException("index " + index + " is not in the format " + simpleDateString); > > String indexDateStr = index.substring(pos + 1); > Long timeLong; > try { > timeLong = formatter.parse(indexDateStr).getTime(); > } catch (Exception ex) { > throw new IllegalStateException("index " + index + " is not in the format " + simpleDateString + " error:" + ex.getMessage()); > } > String old = sortedIndices.put(timeLong, index); > if (old != null) > throw new IllegalStateException("indices with the identical date are not supported " + old + " vs. " + index); > } > int counter = 1; > Iterator\ indexIter = sortedIndices.values().iterator(); > > while (indexIter.hasNext()) { > String currentIndexName = indexIter.next(); > if (counter \>= maxRollIndices) { > deleteIndex(currentIndexName); > // delete all the older indices > continue; > } > > if (counter == 1) > oldFeedIndexName = currentIndexName; > > if (counter \>= maxSearchIndices) > removeAlias(currentIndexName, searchAlias); > > counter++; > } > } > if(oldFeedIndexName != null) > moveAlias(oldFeedIndexName, newIndexName, feedAlias); > else > addAlias(newIndexName, feedAlias); > > return newIndexName; > } > > public String getSearchIndexName() { > return getIndexName() + "\_search"; > } > > public String getFeedIndexName() { > return getIndexName() + "\_feed"; > } > > public void createIndex(String indexName) { > client.admin().indices().create(new CreateIndexRequest(indexName).settings(createIndexSettings())).actionGet(timeout); > } > > public XContentBuilder createIndexSettings() { > if (createIndexSettings == null) { > try { > createIndexSettings = JsonXContent.contentBuilder().startObject(). > field("index.number\_of\_shards", createIndexShards). > field("index.number\_of\_replicas", createIndexReplicas). > field("index.refresh\_interval", "10s"). > field("index.merge.policy.merge\_factor", 10).endObject(); > } catch (IOException ex) { > throw new RuntimeException(ex); > } > } > return createIndexSettings; > } > > public void deleteIndex(String indexName) { > client.admin().indices().delete(new DeleteIndexRequest(indexName)).actionGet(); > } > > public void addAlias(String indexName, String alias) { > client.admin().indices().aliases(new IndicesAliasesRequest().addAlias(indexName, alias)).actionGet(); > } > > public void removeAlias(String indexName, String alias) { > client.admin().indices().aliases(new IndicesAliasesRequest().removeAlias(indexName, alias)).actionGet(); > } > > public void moveAlias(String oldIndexName, String newIndexName, String alias) { > client.admin().indices().aliases(new IndicesAliasesRequest().addAlias(newIndexName, alias). > removeAlias(oldIndexName, alias)).actionGet(); > } > > public Map\ getAliases(String index) { > Map\ md = client.admin().cluster().state(new ClusterStateRequest()). > actionGet().getState().getMetaData().aliases().get(index); > if (md == null) > return Collections.emptyMap(); > > return md; > } > private static Comparator\ reverseSorter = new Comparator\() { > > @Override > public int compare(Long o1, Long o2) { > return -o1.compareTo(o2); > } > }; > > public String\[\] getConcreteIndices(Set\ set) { > return client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState(). > getMetaData().concreteIndices(set.toArray(new String\[set.size()\])); > } > \`\`\` > > TEST > > \`\`\` > @Test public void rollingIndex() throws Exception { > search.setClient(createTestClient()); > search.setIndexName("tweets"); > String rollIndexTag = search.getIndexName() + "\_roll"; > String searchIndex = search.getIndexName() + "\_search"; > String feedIndex = search.getIndexName() + "\_feed"; > search.rollIndex(4); > assertEquals(1, search.getAliases(rollIndexTag).size()); > assertEquals(1, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(2, search.getAliases(rollIndexTag).size()); > assertEquals(2, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(3, search.getAliases(rollIndexTag).size()); > assertEquals(3, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(4, search.getAliases(rollIndexTag).size()); > assertEquals(4, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(4, search.getAliases(rollIndexTag).size()); > assertEquals(4, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(search.getIndexName(), 4, 3); > assertEquals(4, search.getAliases(rollIndexTag).size()); > assertEquals(3, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > } > \`\`\` Then after flushing it even should be safe to rsync them into another location + get them back. Regards, Peter. ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [December 2, 2011, 11:36pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/13 "2011-12-02T23:36:27Z")
Peter, Yes, rolling off means that the index for a given day has become older than our current online window and is eligible for archiving on tape or another remote location not available to the cluster. So say we're keeping daily indexes for 100 days, on day 101 for an index it can be backed up and sent to tape. Thanks for the pointer, a few questions: 1. Are you using Index Templates with this method? 2. After an index is flushed (and even closed), from where do we reliably copy it and make sure we got all of the needed parts/shards. 3. Are you using this in a production system? Just curious how it's working out. Thanks for the response! --Mike On Fri, Dec 2, 2011 at 3:52 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote: > On 2 Dez., 17:08, Michael Sick [michael.s...@serenesoftware.com](mailto:michael.s...@serenesoftware.com) > wrote: > > > James, > > > > I'm with you on the Backup/Restore. Shay has indicated that it's a high > > priority. We need it to satisfy enterprisy type customers that always > > want > > a stable offsite backup. It would also be a great way to manage > > pushing/pulling time based indexes from a cluster. We're likely to have > > an > > index/day and would like to roll them off the back-end after N days. > > Rolling off means in your case delete from disc or avoid searching on > them? > > Here is some code to do rolling indices: > [Convenient rolling index method · Issue #1500 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1500) > > Then after flushing it even should be safe to rsync them into another > location + get them back. > > Regards, > Peter. ---
**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\ **Post date:** [December 3, 2011, 2:12pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/14 "2011-12-03T14:12:39Z")
On Fri, Dec 2, 2011 at 17:08, Michael Sick [michael.sick@serenesoftware.com](mailto:michael.sick@serenesoftware.com) wrote: > I'm with you on the Backup/Restore. Shay has indicated that it's a high > priority. We need it to satisfy enterprisy type customers that always want a > stable offsite backup. I setup a production setup of ES that does tens of millions of queries per day. And I solve this by not having ES be the primary datastore for anything, it's just treated as a specialized index. I.e. the primary datastore is data scattered through various RDMSs, then I have a cronjob that does daily aggregations of all that data into a flat daily rotating table that'll become the Elasticsearch index. Then to populate the index I effectively do a SELECT \* from that table and inject into a new daily ES index via the bulk api. This means that: - In an organization that's used to managing production data via RDMSs there's no new store of production data, just a specialized index. - The ES index can be nuked at any time and we can resume search operations in the time it would take to run that SELECT \* \> ES cronjob. Currently that's around 10 minutes. - We don't have to set up anything new to backup / manage the data. E.g. we have a regular snapshots of production data that are moved to dev environments. The snapshot just copies the RDMSs, and then a cronjob in the dev environment populates the dev Elasticsearch index (which'll by definition by equivalent to production). Now in my case the Elasticsearch dataset isn't that large (it comfortably fits in RAM on one machine), and I only generate new indexes daily, but I don't see any inherent reason for why this strategy couldn't be adapted for larger data / data that's changing all the time. Setting it up like this did a lot to alleviate concerns about introducing new technology in my organization. ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [December 5, 2011, 4:05am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/15 "2011-12-05T04:05:17Z")
I've read the pull request, but I have no experience with ZooKeeper. ZooKeeper uses a fixed list of ZooKeeper nodes, so it’s quite easy for it > to decide if quorum is present or not. Does this comment mean I have to have a few nodes dedicated to just running zookeeper, or does it mean my application nodes are fixed? Because I have no fixed nodes. Amazon manages my instances for me and its services will create new nodes when demand is high, and destroy nodes when demand lessens. I don't know the IPs of these nodes, nor do they hard disks (EBS on AWS). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [December 5, 2011, 7:28pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/16 "2011-12-05T19:28:09Z")
Note that with zookeeper you still have split brains, you just get to a state of no availability when it happens (as far as I know). You can get to similar behavior with the minimum\_master\_nodes setting in elasticsearch discovery (thats not to say that a zookeeper discovery module is not cool). On Mon, Dec 5, 2011 at 6:05 AM, James Cook [jcook@pykl.com](mailto:jcook@pykl.com) wrote: > I've read the pull request, but I have no experience with ZooKeeper. > > ZooKeeper uses a fixed list of ZooKeeper nodes, so it’s quite easy for it > > > to decide if quorum is present or not. > > Does this comment mean I have to have a few nodes dedicated to just > running zookeeper, or does it mean my application nodes are fixed? Because I have > no fixed nodes. Amazon manages my instances for me and its services will > create new nodes when demand is high, and destroy nodes when demand > lessens. I don't know the IPs of these nodes, nor do they hard disks (EBS > on AWS). ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [December 6, 2011, 7:45pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/17 "2011-12-06T19:45:14Z")
Can you get to "no availability" using minimum\_master\_nodes when you have a totally dynamic collection of nodes? (I don't know how many will be created/destroyed by external manager to handle load.) ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [December 9, 2011, 5:58pm UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/18 "2011-12-09T17:58:21Z")
A little late on this thread, but figured I'd share my experience. We were able to replace two enterprise search systems. One was a legacy that we wrote a emulation layer on top of to act as a drop in replacement. The other search system was costing way too much money and the level of support for issues I ran into was very poor, even with harassing people on a daily basis, and the performance wasn't that good, after jumping through some hoops on my side to optimize. We compared elasticsearch to solr back in fall of 2010 and at that time elasticsearch had many compelling features that differentiated it from Solr. Without tuning anything, elasticsearch was 10x faster. I actually assumed by tests were broken. Now, I could probably have gotten solr to the same performance level, but why go through the effort? In summary: - elasticsearch saved my company probaly 50K / year - It improved performance from the systems I replaced by 10x - Enabled lots of new features we didn't previously had - Shay and others on the discussion groups provide a great level of support. - Scales horizontally... just throw new servers into the cluster to add capacity We've had a couple of hiccups around network partitions. Early versions could nuke some data. 0.16 fixed most of these issues, but we still had a few indices corrupted on this release after a major network event. Best Regards, Paul On Nov 30, 3:33 am, Vitalii Siryi [vitalii.si...@gmail.com](mailto:vitalii.si...@gmail.com) wrote: > I want to convince my customer to use theES. To do this I needsuccessstories aboutESusageonproduction. I tried to find similar > information at the Elasticsearch official site and in Google but not > found. > > I'll be very happy if you write how many documents you are indexing > and how many approximate number of request is served daily. Some links > toproductionapplication will be very helpful. > > Thanks! > > Best regards, Vitalii Siryi ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:45am UTC](https://discuss.elastic.co/t/please-tell-about-the-success-story-about-es-usage-on-production/6022/19 "2017-07-06T03:45:56Z")