Hi all,
I'm running into this error message when running extensive queries from a python script and when adding / removing nodes. It seems to occur only when putting elasticsearch "to work". There is no problem when running the normal workload.
My use case:
I'm logging time series data, one index per day per source.
Currently we mostly have 4 to 8 sources online at a time.
The data consists of one document per second, the document contains about 20 metrics that get mapped to int's, doubles, ....
This setup has been running for about a year, one index is mostly 40 - 60 MB, the total indexes now is 1200, each index has 3 primaries and 2 replica's, total store size is now 210 Gigs (including replica's)
My hardware:
Now: 3x quad-core vps running on (local)ssd with 16 Gigs of memory of which elastic gets 8.
previous: 3x quad-core dedicated on spinning disks with the same memory settings.
Both setups produce the same problem.
The software:
Elastic 6.5.1 running on ubuntu 18.04 with openjdk 1.8.0_191, nofile is set to 200000:200000, swap disabled.
2 questions:
-
How do I resolve this error (..duh..) Last time I ran the script the cluster came to a grinding halt and could only recover after rebooting with 5 nodes.. with 3 it crashed before reallocating shards with the same error.
-
What would be a better scheme of saving this data ? Are there to many small indexes now ? Maybe it's better to combine the indexes per week / month / year ?
There is a small portion of the logs here: https://pastebin.com/Y8ftm6uy Please let me know if you require more logs / other data.
Thanx for any insight !!