_field_caps API takes long time (more than a minute) to return results causing kibana to not be able to refresh index pattern

Elastic Stack Elasticsearch
1

Elasticsearch version 6.7.2

JVM version Java HotSpot(TM) 64-Bit Server VM (1.8.0_172):

OS version Linux

field_caps API takes long time (more than a minute) to return results :

Steps to reproduce :
1.) Index pattern with about 409 matching indices with about 61 fields
2.) curl -XGET https://esurl:9200/indexpattern-/_field_caps?fields=
2.) Takes 1 minute 45 seconds to get the result.

This causes kibana to not be able to refresh the index pattern as default timeout is set to 30 seconds.
Is it on expected lines , Anything that can be done to speed up this api ?. Cluster is not resource constrained?

I found a similar issue Kibana index is unresponsive
with responses by @chrisronline

2

Hi @nikhil-bhat,

To get a ball bark idea of the impact, I attempted to run GET _field_caps?fields=* -- which should be rather expensive -- on a moderate 6.7 cloud cluster with 268 indices, most of them generated by metricbeat with 1342 fields, and the response took about 1s.

Are you certain that your cluster is not under pressure?

Rgs,
Dimitris

3

hi @dliappis hope you are having a pleasant day .
the cluster in question is not under any load .

the field caps api still took above 1minute for the index pattern(415 indices) with 71 fields .
should i attach node stats and any other detail for emphasis.
All of the indices are of size less than 30 mb .

4

Hey @nikhil-bhat,

Could you collect the output from the hot_threads API during the execution of your field caps query and paste here the top entries? Please take care that you mask any private data, such as public IP addresses, before posting here.

Dimitris

5

here are the links to hot thread api output after i hit refresh on index pattern in kibana

HIT 1

HIT 2

6

Hello @nikhil-bhat,

Looking at your hot threads output first of all we can see that some of your data nodes (data-04/08/03/01) are heavily occupied in indexing.

A number of the thread dumps from data nodes though (data-01/data-05) show that SearchGuard is installed and it seems to spend most of the time resolving index patterns while TransportFieldCapabilitiesAction is being called.

Since SearchGuard is a third party plugin we can’t really advise any further about what’s going on here. If the performance issue can be reproduced with Search Guard disabled or when using the free security features from Elastic included in 6.8+, we're happy to continue investigating.

Regards,
Dimitris

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.