we are facing a design problem in the place I work. Let's say that we have many data structures that contain documents, this data structure can be of different forms, with fields that are generated by the user of a data structure. Then, inside this data structure, there are the documents inserted that are also indexed and searched.
data structure A: - name: string - id: string - date_of_birth: date Data strcuture B: - name: string - id: integeger - age: integer
now, data A and data B can contain from 0-N documents.
Should we create an index per data structure? or try to fill all data structures in a big index? or index per data type? (thus create an index per each data type, that's having ~10 indexes) or what else?
So far we used an index per data structure, but we end up having a huge amount of indexes and shard in a single node >1000 indexes (is there any limitation or suggestion in the number of index per node? note that the queries are never cross-indexes). so far the queries are rather fast, creating an index takes a lot of time (±10 sec)
having a big index is problematic because any new structure we add fields and having many fields in an index is not good (is written somewhere in the doc, I can't recall the page)
having an index per type of data (one for integer, string etc) may work, but then if the search involves more than a single query it becomes complicated (ex: search over B where id>1 age<20 and name starts with "alb*")
any suggestion or reference to understand that?