# Tuning Elasticsearch - massive virtual memory timeouts **URL:** \ **Category:** Elasticsearch\ **Created:** [December 20, 2018, 3:24pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714 "2018-12-20T15:24:16Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![mathurin68](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Post date:** [December 20, 2018, 3:24pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/1 "2018-12-20T15:24:16Z")
I haven't had time to look at this in awhile and looks like its way out of control - need help/recommendations on tuning this before replicating to another node. I've got winlogbeat indices going back over a year, I've got all kinds of timeouts, etc. PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command 63868 logstash 39 19 13.2G 3372M 21368 S 5.2 1.3 35:41.60 /usr/bin/java -Xms3g -Xmx3g 63818 elasticse 20 0 2.4T 36.8G 18.0G S 5.2 14.6 1:17.63 /usr/bin/java -Xms16g -Xmx16g `curl -X GET "localhost:9200/_cluster/stats?human&pretty"` [https://pastebin.com/bJ2MCrh2](https://pastebin.com/bJ2MCrh2) It's a mess! Who's up for a challenge?????😀 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 20, 2018, 4:26pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/2 "2018-12-20T16:26:41Z")
You have 1800 shards on a single node with 2.3TB of data and 16GB of heap, which averages out as 1.2GB per shard on disk and 112 shards per GB of heap. [This article](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) recommends to aim for shards that are tens of GB in size (20-40GB) and to have ~20 shards per GB of heap. I think this is a simple case of having too many shards with not enough data in them. ---
**Author:** ![mathurin68](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Post date:** [December 21, 2018, 12:29am UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/3 "2018-12-21T00:29:43Z")
OK, cool! Read your link and watched this video - ...and seems like the main problem for most people is too many shards. Currently have a time based index, daily with ~ size of each index 5 GB 3-4,000,000 docs and 10 shards(no idea where the 10 came from) 1800 shards total- and based on what the articles said I should have around 58 shards total right now - 2,300GB(Store Total) /40GB(per shard) First step is to modify current Winlogbeat template to 2 shards going forward(adding a node to the cluster so going with 2) and moving refresh\_interval from 1s to 30. ``` PUT _template/winlogbeat { "template": "winlogbeat-*", "settings": { "number_of_shards": 2, "number_of_replicas": 1, "refresh_interval": "30s" }, "aliases": { "winlogbeat": {} } } ``` Does this look right? And is there a way to shrink more than one index at a time? Thank you for your help!! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 21, 2018, 8:26am UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/4 "2018-12-21T08:26:11Z")
5GB of data per day isn't really enough data to justify daily indices. Weekly indices with 1 shard would be ~35GB according to your figures, which seems about right. Using a replica means that search traffic can be balanced across both nodes, even with a single shard. > [@mathurin68](#): > > And is there a way to shrink more than one index at a time? In the sense of [the shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-shrink-index.html) (i.e. `PUT /index/_shrink`)? I didn't think there was any restriction on calling that API, although there will be some limits in Elasticsearch that mean it queues up some of the shrinks so as not to overload itself. Yet your system is already overloaded, so I would recommend proceeding slowly and not trying to do too many things at once. I would also recommend that you consider using [reindex](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docs-reindex.html) instead, to convert your existing daily indices into weekly ones. I should have mentioned that you can relieve a bunch of heap pressure by [closing](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-open-close.html) any indices you don't need for now, and this will help to make your system more stable. Then you can open a few, reindex them, delete the originals, and repeat, avoiding having all 1800 shards open at once. ---
**Author:** ![mathurin68](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Post date:** [December 21, 2018, 6:22pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/5 "2018-12-21T18:22:01Z")
OK! 1. Moving winlogbeats to weekly indices - In logstash conf.d folder I switched my elasticsearch output from: > index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" to: > index =\> "%{[@metadata][beat]}-%{+xxxx.ww}" ...that seems to work great. 1. Closing open indices for temporary relief Got a status of winlogbeat indices with this: > curl [http://localhost:9200/\_cat/indices/winlogbeat-\*?v](http://localhost:9200/_cat/indices/winlogbeat-*?v) and closed them like this: > curl -X POST "localhost:9200/winlogbeat-2018.01.\*/\_close" ...worked great. 1. QUESTION on moving old indices from daily to weekly: If I reindex like this - > ``` > curl -XPOST "http://alpha:9200/_reindex" -d' > { > "source": { > "index": "winlogbeat-2018.01.*" > }, > "dest": { > "index": "winlogbeat-2018.01" > } > }' > > ``` BUT is this monthly or weekly? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 21, 2018, 6:43pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/6 "2018-12-21T18:43:54Z")
> [@mathurin68](#): > > BUT is this monthly or weekly? That looks monthly: `winlogbeat-2018.01.*` will match all the daily indices from January 2018. There's no easy one-shot way to do this using wildcards, because computers make time too difficult. Maybe monthly is ok for old data, for the sake of simplicity. Alternatively, note that you don't have to do the whole reindex in one single command, so you could work one day at a time: ```plaintext curl -XPOST "http://alpha:9200/_reindex" -d' { "source": { "index": "winlogbeat-2018.01.01" }, "dest": { "index": "winlogbeat-2018.01" } } curl -XPOST "http://alpha:9200/_reindex" -d' { "source": { "index": "winlogbeat-2018.01.02" }, "dest": { "index": "winlogbeat-2018.01" } } ``` etc. Probably best not to kick off too many of these at once. ---
**Author:** ![mathurin68](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Post date:** [December 21, 2018, 7:01pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/7 "2018-12-21T19:01:02Z")
Hmm, OK so I guess there's no way to reindex to weekly? Just to keep everything the same? Thank you so much for your help on this one, definitely learned a lot. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 21, 2018, 7:04pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/8 "2018-12-21T19:04:47Z")
Why not go to monthly indices for older data and adjust the number of primary shards to get a reasonable average shard size? ---
**Author:** ![mathurin68](https://avatars.discourse-cdn.com/v4/letter/m/6bbea6/32.png) [@mathurin68](https://discuss.elastic.co/u/mathurin68)\ **Post date:** [December 21, 2018, 8:04pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/9 "2018-12-21T20:04:33Z")
That would be fine...I was more curious than anything. Ok so reindexing all of winlogbeat-2018.01.\* into winlogbeat-2018.01 BUT one day at a time. Ran this to find out how much January has in it's 'store' > curl [http://localhost:9200/\_cat/indices/winlogbeat-2018.01\*?v](http://localhost:9200/_cat/indices/winlogbeat-2018.01*?v) looks like there's about 133GB in January so 5 shards in winlogbeat-2018.01 should be fine. Starting the process > curl -XPOST "[http://localhost:9200/\_reindex](http://localhost:9200/_reindex)" -H 'Content-Type: application/json' -d' > { > "source": { > "index": "winlogbeat-2018.01.01" > }, > "dest": { > "index": "winlogbeat-2018.01" > } > }' and then curl -XDELETE "[http://localhost:9200/winlogbeat-2018.01.01](http://localhost:9200/winlogbeat-2018.01.01)" This seems to shrink everything drastically. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 21, 2018, 9:22pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/10 "2018-12-21T21:22:58Z")
> [@mathurin68](#): > > I guess there's no way to reindex to weekly? There is, but it's not so easy as the monthly thing: for each day, in turn, reindex it into the appropriate weekly index. You will need to compute the name of the weekly index yourself. Not sure it's worth the hassle. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2019, 9:35pm UTC](https://discuss.elastic.co/t/tuning-elasticsearch-massive-virtual-memory-timeouts/161714/11 "2019-01-18T21:35:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.