Using elasticsearch as framework for other index

Elastic Stack Elasticsearch
1

Hi Shay,

I have a specialized index for binary data (non lucene) which has a
requirement of having at most 500k documents in it, so if there are
more to be indexed i have to creat a new shard.

i was wondering if it is possible to use elasticsearch in this case to
query and index the custom index(s), so that i dont have to program
the sharding and search process, as elasticsearch does this very well.

Are there points in the source code of es where I can jump in to try
this? or is the network and sharding code too
entrenched with the lucene library.

TIA
Andy

2

Hi Andy

I have a specialized index for binary data (non lucene) which has a
requirement of having at most 500k documents in it, so if there are
more to be indexed i have to creat a new shard.

i was wondering if it is possible to use elasticsearch in this case to
query and index the custom index(s), so that i dont have to program
the sharding and search process, as elasticsearch does this very well.

You can't add primary shards to an existing index, as ES uses the number
of shards to calculate where each ID will be stored.

However, an easy solution is to use aliases:

You could just create an alias "binary" and point it to the first real
index (which has a single shard) called "binary_1". When you need to
create more shards, you create a new index "binary_2" and update the
"binary" alias to point to both of them.

That way you can query all your indices as a single "index" while
controlling the number of shards dynamically.

clint