I have cluster consist of 17 nodes. 3 masters, 12 data, 2 coordinating. My cluster consist of approximately 300 indices right now. Each index have 4 shards and set 2 replicas. Some of my indices has between 400-800 gb (there is just few of them) per day (included replicas). My cluster was so slow, often lost some node base on translog already closed error or too many opened files. On some nodes, I configure 409600 max open files. This is openfiles on my cluster:
+----------+----------+----------+----------+
| MAX FD | OPEN FD | DIFF |
+----------+----------+----------+----------+
| 131070 | 510 | 0.39 % | 130560 |
| 409600 | 1187 | 0.29 % | 408413 |
| 409600 | 6018 | 1.47 % | 403582 |
| 409600 | 102349 | 24.99 % | 307251 |
| 409600 | 1961 | 0.48 % | 407639 |
| 409600 | 85900 | 20.97 % | 323700 |
| 409600 | 2439 | 0.60 % | 407161 |
| 409600 | 126252 | 30.82 % | 283348 |
| 409600 | 1891 | 0.46 % | 407709 |
| 131070 | 2112 | 1.61 % | 128958 |
| 131070 | 2159 | 1.65 % | 128911 |
| 131070 | 2256 | 1.72 % | 128814 |
| 131070 | 80871 | 61.70 % | 50199 |
| 131070 | 512 | 0.39 % | 130558 |
| 131070 | 510 | 0.39 % | 130560 |
| 131070 | 496 | 0.38 % | 130574 |
| 131070 | 496 | 0.38 % | 130574 |
+----------+----------+----------+----------+
I think, it is too much openfiles, when i list them with lsof -u elasticsearch, more of them are translogs. I decide to set refresh_interval to -1 and then foceMerge on all old indices older than one day. It helped a lot, but I would like to perform this changes also on big indices more often. So I would like to split them to more indices, eg per 4 or 6 hours. How can I do that? I thought at first use some if statements, but it does not look as a good idea. Then, there is option use ruby filter in logstash. Is that good idea or is that better solution for my problem?