Schema Design in Elastic Search Scenario

(sumitshrestha) #1

We are moving from Hbase centric design to Elastic Search. After a month
long work, We finally moved many of our hbase-centric tables to ES. Our
design is mainly a set of schema's (tables) for different Clients. All
Client have same set of schema(table). So, tables of one client are related
and we have designed accordingly with Client as Index and all tables in it
as that index's type.

But, this design has its problem. Fields with same name in different type
must have same type in ES to support sorting and comparisons. There are
chances that there may be fields with same name in different Table(Type)
with different types. In that case we need to either change their name or
define same field type. Both of them are not easy given large number of
increasing tables that we need to support in each client.

My solution to this is to represent each table as Index and all clients
that needs to have it as type in it. So, each table would have its own
index and one mapping for each index.

But, problem with this design is that logical representation of client is
lost in this case. A client's data will be scattered among different
indices. So, I was in dilemma about a proper design of tables in ES. At ES
side, representing each table as separate index make sense but at Logical
(domain) level, making it as type makes sense.

Can anyone enlighten me over it. Is there are better design possibility.

(system) #2