Tuning index search query, refresh and flush times

Elastic Stack Elasticsearch
1

We currently have a 5 node ElasticSearch cluster running on 2.3.5. Each with 64GB of memory, and 30GB dedicated to ElasticSearch. The messages input is coming from Graylog (2.0.3). Previously, the indices were being written to 1TB mechanical drives, but we have now upgraded all of them to 2TB SSDs. However, we are still seeing unexpectedly high time stats for index search query, refresh and flush times.

Search query 50ms
Refresh 65ms
Flush 1270ms

I've gone through a good amount of tuning documentations, but all my experiments with values so far did not yield notable difference. Can anyone provide some pointers or on what values I should focus my attention on that may be beneficial to our usecase?

2

How are you monitoring ES?

3

Using Elastic HQ plugin, which is essentiall polling the API for stats.indices.search.query_time_in_millis, stats.indices.refresh.total_time_in_milis and stats.indices.flush.total_time_in_millis for the indices.