I'm facing strange performance issues the last days.
Following my development-cluster-setup:
- Single Node
- 12GB RAM
- 4 Cores
- Spinning Disks
- a total of 45Million (~40GB of data) Documents spread across 80 indexes and 80 shards
- Four different daily indexes, with data from 100MB up to 3GB
I allocated 6GB of RAM to JVM Heap.
Because this installation is to get to know with the elk stack, also logstash and kibana are running on that machine.
Now my problem: When opening a Dashboard (Last 3Days) which includes visualisations with cross-index-searches, Elasticsearch is not reachable after a time. Seems like it is hard working, but in top neither CPU, nor RAM or IO-Wait have too high values. The elasticsearch log also doesn't throw any error, but Kibana runs in its timeout (in my case 120s)
Another interesting fact is, that elasticsearch can not be stopped after this freeze and also the query
curl -XGET localhost:9200 returns nothing (it waits for response)
The picture shows the moment when elasticsearch is freezing.
Well, of course this is not much information, but maybe anyone has a hint what I could improve or what I can check to find the error. Maybe the hardware used for this amount of data is too bad?