I don't know your data set here, so please understand these are general recommendations based on experience. Your mileage may vary based on what you're querying, and how much data at once.
This is indicative of memory pressure. Resegmenting your indices to try to avoid memory pressure is not the best way to handle this.
I get heap memory circuit breakers if I use daily indices
This indicates that cluster memory is insufficient for the size of queries you intend to run. There are several ways to deal with this, including using
doc_values in your template, if you haven't already and if your queries are against
not_analyzed strings or numeric values. If your queries are against analyzed strings, or you are still seeing circuit breakers tripped even with
doc_values set, then your best recourse is to increase the amount of cluster memory available. This can be done by increasing the memory on each node (up to a maximum of 30.5G), or by increasing the number of nodes.
One way to improve memory usage without adding hardware would be to start manually mapping the fields you have to use the smallest data-type possible. Elasticsearch guesses with the biggest possible value, so if you send a floating-point value through Logstash, Elasticsearch will consider it a
double, and all integers become
long, unless you map them otherwise. Properly mapping your values can also help avoid circuit breakers because the query won't need to allocate as much memory to aggregate a
short as it will a
As you indicate you're using Logstash, I recommend upgrading to the most recent version, if only for the upgraded default template that ships with Logstash, because it applies
doc_values by default as much as possible.
... query buffer overrun if I use hourly
This is likely because hourly indices add so many shards that your cluster is having some issues managing them all, or querying across all of them at once. Just because Elasticsearch doesn't immediately complain about the shard count doesn't mean that too many won't cause issues. Ideally, to have no pressure with shards, you should not have more than 400 - 700 total shards on a 30G node, with only a few of those being active (currently indexing). Memory pressure counts here too as each active shard wants 250M of 10% of the heap allocated to the index buffer (
indices.memory.index_buffer_size, if you want to customize the amount set aside from the heap). Inactive shards each want 4M of that 10%. I say want, because if there are too many shards on a node, Elasticsearch will reduce the amount available to all shards to allow them all to fit. On a node with 30G of RAM, that means that 3G is allocated by default. If all shards were inactive, that would mean a safe fit for 774 shards on that server, but any active shards reduce that count quickly. This isn't to say you can't have a box with 1000 shards, but more often than not, we see similar problems with indexing and search performance on boxes with high shard counts.
Again, these are general recommendations I offer as the things you have shared indicate memory pressure. And the best way to fix memory pressure is not to subdivide your indices in ever more complex ways, as that only masks the problem and adds complexity to your queries.