Search perfomance using many indices with one alias

Elastic Stack Elasticsearch
1

Hey there,

we run into following problem, maybe you can give us some adivce or best practice:

We use many indices (80) and for search we use one alias to query all of them at the same time as search results can be in found in any index and should be returned together.

The reason for this many indices is that every index has different fields, so using only one index (e.g. with different types) would result in sparsity and a huge number of fields ( > 1.000) (which is not recommented by Elastic).

However, with using many indices our search perfomance is very poor (around 5 - 10 seconds).

Any advice on this? Is it bad to use so many indices and adress all together in one search? Any other hints where to look to improve search performance?

Thanks,
Julia

2

Are you searching on a single type or across multiple types? When searching on multiple types, the search fields needs to be present on all the types.

Also try limiting the number of fields returned as part of search response, using "_source" in your query.

Using an alias to query multiple indices shouldn't impact search performance. You need to check if you have proper search analyzers on the fields.

3

Hey,

we have one type + 2 child-types per index. Every "type" on every index has a few fields in common and many different fields, that's the reason why we split it into many indices instead of using only one index.

Do you mean, even then search across multiple indices it would be wise to have all fields present? When searching, all field names are always part of the query even if they do not exist in the particular index. Is this bad?

Thanks,
Julia

4

Hi Jul

As long as you have the common fields across Types, which are part of your search query you should be good. For e.g. Type A in Index 1 can have 20 fields and Type B in Index 2 can have 10 fields. But there should be common set of fields across Types (A & B) across indices (Index 1 & 2) which you are searching on.

Are queries being slow or is there a particular query which is slow?

Also check the filters in your query.

Can you please elaborate on this?

5

Hey,

where are only very few common fields in Type A and Type B.

Up to now, the search query contained a query string query with all fields names of Type A, B, ... (field1, field2, field3, ...).

If I use "field*" instead, the query perfoms a lot better, but I cannot use this approach (restrictions based on users).

I just found the Indices Query (https://www.elastic.co/guide/en/elasticsearch/reference/5.4/query-dsl-indices-query.html) which looks quite promising as I know in advance which field belongs to with type.
Anyhow, as it deprecated, I'd like to build the query correctly but did not (yet) find the correct way to use it. Any suggestions?

Thanks,
Julia

6

Try specifying the fields on which the search should happen and see the results. For e.g. if you want to search on fields 1,2,3 from Type A and fields 4,5,6 from Type B specify them in query like
[type1.field1, type1.field2,type2.field4] etc.

Refer to multi match query:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-multi-match-query.html

If you search for a field which is not present in the type, Elastic Search will still look for it, as it has a schema less model.

7

Hey code_blue,

thanks for your hints. Especially the one about schema less model helps a lot to understand how the queries are performed.

A term query with type1.field1 does not yield any results:

{
  "query": {
    "query_string": {
      "default_field": "type1.field1",
      "query": "test"
    }
  }
}

I found a very well perfoming solution using IndicesQuery, but it is deprecated. I opened a new topic for my question as it might be relevant to other users as well: Search performance: IndicesQuery (deprecated) vs. query on field _index

Jul

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.