I am trying to understand what is causing this. There was a increase in search latency and query latency during some part of time on our cluster causing some reports to fail. On looking into some cluster stats, I observed there was an increase in the number of active shards for a brief amount of time as seen in the graph. I want to understand why this is happening . Also a couple of nodes were gc-ing at the same time. Could you please help me understand why the shards increased and dropped back to normal.
Hi,
this might be hard to tell after the fact. Maybe it has been caused by a maintenance script that created a new index and cleaned up old ones a bit later? You could check the cluster logs whether you find log lines around that time, similar to:
[2019-07-26T11:16:16,792][INFO ][o.e.c.m.MetaDataCreateIndexService] [io] [twitter] creating index, cause [api], templates [], shards [1]/[1], mappings []
If you want to dig deeper via the command line you can use the various "cat" APIs, in that case you'd probably want to use the cat shards API to get a list of the current shards at that point in time in the cluster. This means, if you want to see changes over time you'd need to call that API periodically.
Daniel
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.