How to create sub-indices inside indices

Elastic Stack Elasticsearch
1

I have currently around 200 customer databases under one ES node and for each database, I am running 10 different queries to assess performance. For that I have 200*10 indices now, but that does not look good. I plan to create one index per customer and create sub indices under each for each query result. Is that feasible?

2

Sub indices does not exist.
But you can put all data within the same index and just filter data by user id.

3

So can I create one index per customer and filter the queries by type ID or query ID? Because the query results yield similar data format from each database and I want to compare them in Kibana

4

No you create one index for all. Then filter by customer.
You can also use filtered aliases and create one alias per customer then.

1 Like
5

Hello again, I just noticed something today that Kibana is not showing more than 100 indices in the "Discover Indices" tab although running this query
GET _search
{
"query": {
"match_all": {}
}
}
displays more.
Since you advised me to create a single index, I have 2 questions -

  1. Is there any restriction on the number of types too? E.g if I have 500 types within 1 large index, will Kibana display only 100?
  2. Will Elastic, in the future, impose a restriction on the number of indices created and/or increase the default size of each one?
    Sorry for bugging you but these info will be important to me for designing the node.
6

In Elasticsearch 6.0 onwards only a single type per index is allowed. You can however add a field to your documents indicating type and use this for filtering as long as there are no mapping conflicts.

Having lots of small indices and shards is inefficient as there is overhead for each shard. Having an index per customer works for few users but does not scale well. From Elasticsearch 7.x i believe there will be a soft limit for the number of shards per node.

7

Thanks for the explanation. But the procedure is becoming quite confusing.
Firstly the "document_type" field has been deprecated and now I have learnt today that only a single type per index is allowed. My question is what is the purpose of "type" then?
So can I modify the number of indices or types shown by Kibana or that is restricted too? Creating index patterns and segregating fields in Kibana gets quite confusing.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.