We are observing the query rate on our ES cluster. The stat we are looking at is "nodes.indices.search.query_total" under "GET /_nodes/stats/indices/search".
Here is how the metric looks on our cluster, with the query count on each node stacked one on top of the other. (An aside: the cluster had 8nodes earlier ... which was reduced to 6 nodes)
In the beginning the query count used to be pretty flat. We started running curator at 7:30am and now we notice that the query count shoots up as soon as curator runs and keeps rising until midnight (even though curator finishes running in few minutes). At midnight, query count flattens out until curator is run again (next day at 7:30am).
Here is an alternate view of the same metric viewed as a "rate" i.e. queries_per_second:
Normally queries/sec is around 2 queries/sec. However, it goes up to 15queries/sec whenever we run curator and stay up until midnight. At midnight it becomes 2qps and stays there until we run curator again.
Any ideas why/how curator could affect query rate?
The actions in curator yml are: 1. Delete indices older than a week 2. Create snapshot for yesterday's index 3. Delete old snapshots (which is a no op right now as we do not have such old snapshots). The complete actions.yml at the bottom.
Any suggestions on how we can chase this would be greatly appreciated!
actions: 1: action: delete_indices description: >- Delete indices older than 8 days (based on index name), for 'filebeat-' prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True disable_action: False filters: - filtertype: pattern kind: prefix value: filebeat- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 8 2: action: snapshot description: >- Snapshot 'filebeat-' prefixed indices older than 1 day (based on index creation_date). Wait for the snapshot to complete. Do not skip the repository filesystem access check. Use the other options to create the snapshot. options: repository: s3_repository name: filebeat-%Y-%m-%d-%H-%M-%S ignore_unavailable: False include_global_state: True partial: False wait_for_completion: True skip_repo_fs_check: False disable_action: False filters: - filtertype: pattern kind: prefix value: filebeat- - filtertype: age source: creation_date direction: older unit: days unit_count: 1 - filtertype: age source: creation_date direction: younger unit: days unit_count: 2 3: action: delete_snapshots description: >- Delete snapshots from the selected repository older than 21 days (based on creation_date), for 'filebeat-' prefixed snapshots. options: repository: s3_repository disable_action: False filters: - filtertype: pattern kind: prefix value: filebeat- exclude: - filtertype: age source: creation_date direction: older unit: days unit_count: 22