# Has anyone had to tackle effective dating? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 12, 2012, 9:51pm UTC](https://discuss.elastic.co/t/has-anyone-had-to-tackle-effective-dating/6370 "2012-01-12T21:51:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Roly\_Vicaria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roly_vicaria/32/506_2.png) [@Roly\_Vicaria](https://discuss.elastic.co/u/Roly_Vicaria)\ **Post date:** [January 12, 2012, 9:51pm UTC](https://discuss.elastic.co/t/has-anyone-had-to-tackle-effective-dating/6370/1 "2012-01-12T21:51:47Z")
We're using MongoDB as our primary data store and using a river plugin to get data flowing into ES. Question we're facing now is how to tackle effective dating. We figure that we could ignore it and only push current docs into ES. This would force us to drop our indices every night (midnight sounds good) and rebuild them with all the currently effective docs. I'm curious if ES has any mechanism that could assist. I saw that there are facilities for timestamps, ttl, and versioning. It feels like those features could help solve the problem on the ES side, but the solution isn't obvious to me. Anyone come across this problem? ---
**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\ **Post date:** [January 13, 2012, 5:44am UTC](https://discuss.elastic.co/t/has-anyone-had-to-tackle-effective-dating/6370/2 "2012-01-13T05:44:12Z")
"Dating" questions asked in ES forum .... that is something new 😃 ~Vineeth On Fri, Jan 13, 2012 at 3:21 AM, Roly Vicaria [rolyv19@gmail.com](mailto:rolyv19@gmail.com) wrote: > We're using MongoDB as our primary data store and using a river plugin to > get data flowing into ES. Question we're facing now is how to tackle > effective dating. We figure that we could ignore it and only push current > docs into ES. This would force us to drop our indices every night (midnight > sounds good) and rebuild them with all the currently effective docs. I'm > curious if ES has any mechanism that could assist. I saw that there are > facilities for timestamps, ttl, and versioning. It feels like those > features could help solve the problem on the ES side, but the solution > isn't obvious to me. Anyone come across this problem? ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [January 13, 2012, 9:51am UTC](https://discuss.elastic.co/t/has-anyone-had-to-tackle-effective-dating/6370/3 "2012-01-13T09:51:33Z")
> "Dating" questions asked in ES forum .... that is something new 😃 Personal dating we cannot solve here 😉 Regarding your software problem, yes check out ttl and also how to use a rolling index: > > > Here is some code where a rolling index pattern is implemented. Imagine you have… a logical index named 'tweets', now you want to create every day a new index to keep the indices small (Its a better scalable 'sharding', but only if you have time dependent data). Now, in the proposed code you will have to call rollIndex(maximumIndices) once a day. > > Then the new indices are all 'tagged' with a 'tweets\_roll' alias (for later retrieval -\> imrovable?), there is a group of indices for searching (tweets\_search) and feeding (tweets\_feed). Per default it creates a search alias on all indices and a feed alias only for the very latest. It separates the search and the roll alias as it could be the case that one wants to keep some older indices but do not want to search on it. > > What do you think? Its rather simple but it works - a simple test below the code. > > \`\`\` > private static final String simpleDateString = "yyyy-MM-dd-HH-mm-ss"; > public String rollIndex(int maxRollIndices) { > return rollIndex(getIndexName(), maxRollIndices, maxRollIndices); > } > > public String rollIndex(String indexName, int maxRollIndices, int maxSearchIndices) { > String rollAlias = indexName + "\_roll"; > SimpleDateFormat formatter = new SimpleDateFormat(simpleDateString); > > if (maxRollIndices \< 1 || maxSearchIndices \< 1) > throw new RuntimeException("remaining indices, search indices and feeding indices must be at least 1"); > > // get old aliases > Map\ allRollingAliases = getAliases(rollAlias); > > // always create new index and append aliases > String searchAlias = getSearchIndexName(); > String feedAlias = getFeedIndexName(); > String newIndexName = indexName + "\_" + formatter.format(new Date()); > > createIndex(newIndexName); > addAlias(newIndexName, searchAlias); > addAlias(newIndexName, rollAlias); > String oldFeedIndexName = null; > > if (allRollingAliases.isEmpty()) { > // do nothing for now > } else { > TreeMap\ sortedIndices = new TreeMap\(reverseSorter); > String\[\] concreteIndices = getConcreteIndices(allRollingAliases.keySet()); > //logger.info("aliases:" + allRollingAliases + ", indices:" + Arrays.toString(concreteIndices)); > for (String index : concreteIndices) { > int pos = index.indexOf("\_"); > if (pos \< 0) > throw new IllegalStateException("index " + index + " is not in the format " + simpleDateString); > > String indexDateStr = index.substring(pos + 1); > Long timeLong; > try { > timeLong = formatter.parse(indexDateStr).getTime(); > } catch (Exception ex) { > throw new IllegalStateException("index " + index + " is not in the format " + simpleDateString + " error:" + ex.getMessage()); > } > String old = sortedIndices.put(timeLong, index); > if (old != null) > throw new IllegalStateException("indices with the identical date are not supported " + old + " vs. " + index); > } > int counter = 1; > Iterator\ indexIter = sortedIndices.values().iterator(); > > while (indexIter.hasNext()) { > String currentIndexName = indexIter.next(); > if (counter \>= maxRollIndices) { > deleteIndex(currentIndexName); > // delete all the older indices > continue; > } > > if (counter == 1) > oldFeedIndexName = currentIndexName; > > if (counter \>= maxSearchIndices) > removeAlias(currentIndexName, searchAlias); > > counter++; > } > } > if(oldFeedIndexName != null) > moveAlias(oldFeedIndexName, newIndexName, feedAlias); > else > addAlias(newIndexName, feedAlias); > > return newIndexName; > } > > public String getSearchIndexName() { > return getIndexName() + "\_search"; > } > > public String getFeedIndexName() { > return getIndexName() + "\_feed"; > } > > public void createIndex(String indexName) { > client.admin().indices().create(new CreateIndexRequest(indexName).settings(createIndexSettings())).actionGet(timeout); > } > > public XContentBuilder createIndexSettings() { > if (createIndexSettings == null) { > try { > createIndexSettings = JsonXContent.contentBuilder().startObject(). > field("index.number\_of\_shards", createIndexShards). > field("index.number\_of\_replicas", createIndexReplicas). > field("index.refresh\_interval", "10s"). > field("index.merge.policy.merge\_factor", 10).endObject(); > } catch (IOException ex) { > throw new RuntimeException(ex); > } > } > return createIndexSettings; > } > > public void deleteIndex(String indexName) { > client.admin().indices().delete(new DeleteIndexRequest(indexName)).actionGet(); > } > > public void addAlias(String indexName, String alias) { > client.admin().indices().aliases(new IndicesAliasesRequest().addAlias(indexName, alias)).actionGet(); > } > > public void removeAlias(String indexName, String alias) { > client.admin().indices().aliases(new IndicesAliasesRequest().removeAlias(indexName, alias)).actionGet(); > } > > public void moveAlias(String oldIndexName, String newIndexName, String alias) { > client.admin().indices().aliases(new IndicesAliasesRequest().addAlias(newIndexName, alias). > removeAlias(oldIndexName, alias)).actionGet(); > } > > public Map\ getAliases(String index) { > Map\ md = client.admin().cluster().state(new ClusterStateRequest()). > actionGet().getState().getMetaData().aliases().get(index); > if (md == null) > return Collections.emptyMap(); > > return md; > } > private static Comparator\ reverseSorter = new Comparator\() { > > @Override > public int compare(Long o1, Long o2) { > return -o1.compareTo(o2); > } > }; > > public String\[\] getConcreteIndices(Set\ set) { > return client.admin().cluster().state(new ClusterStateRequest()).actionGet().getState(). > getMetaData().concreteIndices(set.toArray(new String\[set.size()\])); > } > \`\`\` > > TEST > > \`\`\` > @Test public void rollingIndex() throws Exception { > search.setClient(createTestClient()); > search.setIndexName("tweets"); > String rollIndexTag = search.getIndexName() + "\_roll"; > String searchIndex = search.getIndexName() + "\_search"; > String feedIndex = search.getIndexName() + "\_feed"; > search.rollIndex(4); > assertEquals(1, search.getAliases(rollIndexTag).size()); > assertEquals(1, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(2, search.getAliases(rollIndexTag).size()); > assertEquals(2, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(3, search.getAliases(rollIndexTag).size()); > assertEquals(3, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(4, search.getAliases(rollIndexTag).size()); > assertEquals(4, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(4); > assertEquals(4, search.getAliases(rollIndexTag).size()); > assertEquals(4, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > > Thread.sleep(1000); > search.rollIndex(search.getIndexName(), 4, 3); > assertEquals(4, search.getAliases(rollIndexTag).size()); > assertEquals(3, search.getAliases(searchIndex).size()); > assertEquals(1, search.getAliases(feedIndex).size()); > } > \`\`\` Peter. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:42am UTC](https://discuss.elastic.co/t/has-anyone-had-to-tackle-effective-dating/6370/4 "2017-07-06T03:42:50Z")
Closed on July 6, 2017, 3:42am UTC