Index updating/ refreshing for parallel applications

Elastic Stack Elasticsearch
1

Hello,

we use elasticsearch now since version .11 and we really like it (currently
on .19.11). However, one thing thats a big problem for us is to update the
index while it is still queried by other requests.

We currently do a index-shift using alias in a java singleton e.g.:

private static String searchindex = "searchindex"; <- is only alias
private static String indexingIndexA = "indexingindex_a"; <- is a real index
private static String indexingIndexB = "indexingindex_b"; <- is a real index

String workingIndex;
String currentActiveSearchIndex;
if (singletonBean.getAliasTo().equals(indexingIndexA)) {
singletonBean.setAliasTo(indexingIndexB);
workingIndex = indexingIndexB;
currentActiveSearchIndex = indexingIndexA;
} else {
singletonBean.setAliasTo(indexingIndexA);
workingIndex = indexingIndexA;
currentActiveSearchIndex = indexingIndexB;
}

    try {
        client.admin().indices().delete(new

DeleteIndexRequest(workingIndex)).actionGet();
} catch (Exception e) {
logger.info(e.getMessage());
}

... do index work ....

client.admin().indices().refresh(new RefreshRequest(workingIndex));
client.admin().indices().aliases(new
IndicesAliasesRequest().removeAlias(currentActiveSearchIndex,
searchindex).addAlias(workingIndex, searchindex));

This runs quite well as long as its the only one that works on it. Now we
wanted to call ES from a second and third application and get the index
refreshed, however when in same time a request is done to the same index it
will lead to trouble as the alias swap can occure right in the middle.

How can we upgrade the whole index from any app without breaking the
ongoing requests of the other ones?
In solr there exists a "swap(index, index)" command where 2 indexes get
swapped in way that all requests prior to time x get done by first and
after that by second index, meaning that solr will handle the parallel
request problems.

Any help would be really appreciated.

Best Regards,

KB

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Equivalent of swap(index, index) in elasticsearch would be this:

curl -XPOST 'http://localhost:9200/_aliases' -d '
{
"actions" : [
{ "remove" : { "index" : "indexingindex_a", "alias" :
"indexing_index" } },
{ "remove" : { "index" : "indexingindex_b", "alias" :
"searching_index" } },
{ "add" : { "index" : "indexingindex_a", "alias" :
"searching_index" } },
{ "add" : { "index" : "indexingindex_b", "alias" : "indexing_index"
} }
]
}'

I am not sure how this (or solr's swap) would help you though. Even if you
will somehow make refresh with swap an atomic operation, you will still
have problems If one application will refresh and swap indices while
another applications is trying to index. It sounds like you need some sort
of cluster-wide lock that all other applications would observe. Maybe you
can create a special index with a record in it that would contain the id of
the process that currently "owns" the working index.

On Thursday, April 18, 2013 5:47:05 AM UTC-4, K.B. wrote:

Hello,

we use elasticsearch now since version .11 and we really like it
(currently on .19.11). However, one thing thats a big problem for us is to
update the index while it is still queried by other requests.

We currently do a index-shift using alias in a java singleton e.g.:

private static String searchindex = "searchindex"; <- is only alias
private static String indexingIndexA = "indexingindex_a"; <- is a real
index
private static String indexingIndexB = "indexingindex_b"; <- is a real
index

String workingIndex;
String currentActiveSearchIndex;
if (singletonBean.getAliasTo().equals(indexingIndexA)) {
singletonBean.setAliasTo(indexingIndexB);
workingIndex = indexingIndexB;
currentActiveSearchIndex = indexingIndexA;
} else {
singletonBean.setAliasTo(indexingIndexA);
workingIndex = indexingIndexA;
currentActiveSearchIndex = indexingIndexB;
}

    try {
        client.admin().indices().delete(new

DeleteIndexRequest(workingIndex)).actionGet();
} catch (Exception e) {
logger.info(e.getMessage());
}

... do index work ....

client.admin().indices().refresh(new RefreshRequest(workingIndex));
client.admin().indices().aliases(new
IndicesAliasesRequest().removeAlias(currentActiveSearchIndex,
searchindex).addAlias(workingIndex, searchindex));

This runs quite well as long as its the only one that works on it. Now we
wanted to call ES from a second and third application and get the index
refreshed, however when in same time a request is done to the same index it
will lead to trouble as the alias swap can occure right in the middle.

How can we upgrade the whole index from any app without breaking the
ongoing requests of the other ones?
In solr there exists a "swap(index, index)" command where 2 indexes get
swapped in way that all requests prior to time x get done by first and
after that by second index, meaning that solr will handle the parallel
request problems.

Any help would be really appreciated.

Best Regards,

KB

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.