So i did discover something interesting after this post.
It is only in kibana the query times out, but if I do a direct elasticsearch query, it returns in under the timeout.
The use case is "someone doesn't know what field the data is in" and needs to do a quick query to find the data so they can filter down to the index. I find it very odd that it is both metricbeat and filebeat exhibiting the problem, and the two different clusters have different amount of data.
2 different clusters are exhibiting the same behavior, 3 different sets of indices in each cluster exhibit the same problem, all are set to 1 shard with 1 replica
Data nodes are 32gb ram, 16vcpu, 1tb of disk (c5.4xlarge)
1 is 24 data nodes . 929 indices.
2 is 44 data nodes. 910 indices
It doesn't feel like the 30 second timeout is really the issue, it feels like something else is wrong with those indices in being queried by kibana, this is happening even when I just search the two individual index patterns of metricbeat and filebeat.