# Memory leak in NioClientSocketChannel? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 18, 2011, 10:24am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881 "2011-07-18T10:24:53Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [July 18, 2011, 10:24am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/1 "2011-07-18T10:24:53Z")
Hi, Our application was running out of memory and I was investigating the issue. It tourned out that elasticsearch was keeping over 600 Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, which aren't cleaned by the GC. I analyzed the heap in MAT but coultdn't find any referenced data (see screenshot).Is this due to a NIO buffer which isn't read out completely and is kept in memory? strvalConnected is also set to false, thus the memory should get reclaimed. I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any ideas what might cause this? Thanks, Thibaut ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 18, 2011, 4:57pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/2 "2011-07-18T16:57:45Z")
Can you dig down a bit more and see where its being used? On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz \< [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote: > Hi, > > Our application was running out of memory and I was investigating the > issue. It tourned out that elasticsearch was keeping over 600 > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > which aren't cleaned by the GC. > > I analyzed the heap in MAT but coultdn't find any referenced data (see > screenshot).Is this due to a NIO buffer which isn't read out > completely and is kept in memory? strvalConnected is also set to > false, thus the memory should get reclaimed. > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > ideas what might cause this? > > Thanks, > Thibaut ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [July 18, 2011, 8:40pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/3 "2011-07-18T20:40:54Z")
Hi, the data is being kept in the buffer array of the BigEndianHeapChannelBuffer instances in jetty (see screenshot). at org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer org.elasticsearch.common.netty.buffer.DynamicChannelBuffer org.elasticsearch.transport.netty.SizeHeaderFrameDecoder org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext org.elasticsearch.common.netty.channel.DefaultChannelPipeline Thanks, Thibaut On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Can you dig down a bit more and see where its being used? > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > Hi, > > > > Our application was running out of memory and I was investigating the > > issue. It tourned out that elasticsearch was keeping over 600 > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > > which aren't cleaned by the GC. > > > > I analyzed the heap in MAT but coultdn't find any referenced data (see > > screenshot).Is this due to a NIO buffer which isn't read out > > completely and is kept in memory? strvalConnected is also set to > > false, thus the memory should get reclaimed. > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > > ideas what might cause this? > > > > Thanks, > > Thibaut ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 18, 2011, 10:01pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/4 "2011-07-18T22:01:38Z")
Are you sending large messages around? For example, large bulk requests, asking for a very large number of hits in one request? On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz \< [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote: > Hi, > > the data is being kept in the buffer array of the > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > at > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > Thanks, > Thibaut > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Can you dig down a bit more and see where its being used? > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > Hi, > > > > > > Our application was running out of memory and I was investigating the > > > issue. It tourned out that elasticsearch was keeping over 600 > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > > > which aren't cleaned by the GC. > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data (see > > > screenshot).Is this due to a NIO buffer which isn't read out > > > completely and is kept in memory? strvalConnected is also set to > > > false, thus the memory should get reclaimed. > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > > > ideas what might cause this? > > > > > > Thanks, > > > Thibaut ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [July 19, 2011, 10:19am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/5 "2011-07-19T10:19:03Z")
Hi, I'm scrolling over a scan: SearchRequestBuilder srb = searchclient.getClient().prepareSearch(indexnames.toArray(new String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); I only requested the keys, but just saw that I was requesting 10000 hits per shard. I have over \> 50 shards, but not all of them return results. But I assume I got about 200000 results per scroll. I limited that. Changing the cached Pool to a thread pool which releases the threads after a few seconds should fix this? Thanks, Thibaut On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Are you sending large messages around? For example, large bulk requests, > asking for a very large number of hits in one request? > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > Hi, > > > > the data is being kept in the buffer array of the > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > at > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > Thanks, > > Thibaut > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Can you dig down a bit more and see where its being used? > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > Hi, > > > > > > > > Our application was running out of memory and I was investigating the > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > > > > which aren't cleaned by the GC. > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data (see > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > completely and is kept in memory? strvalConnected is also set to > > > > false, thus the memory should get reclaimed. > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > > > > ideas what might cause this? > > > > > > > > Thanks, > > > > Thibaut ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 19, 2011, 6:58pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/6 "2011-07-19T18:58:52Z")
Heya, On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz \< [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote: > Hi, > > I'm scrolling over a scan: > > SearchRequestBuilder srb = > searchclient.getClient().prepareSearch(indexnames.toArray(new > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > I only requested the keys, but just saw that I was requesting 10000 > hits per shard. I have over \> 50 shards, but not all of them return > results. But I assume I got about 200000 results per scroll. I limited > that. > > Changing the cached Pool to a thread pool which releases the threads > after a few seconds should fix this? No, thats not how netty works..., started a discussion with Trustin to see if we can improve on it. > Thanks, > Thibaut > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Are you sending large messages around? For example, large bulk requests, > > asking for a very large number of hits in one request? > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > Hi, > > > > > > the data is being kept in the buffer array of the > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > at > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > Thanks, > > > Thibaut > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > Our application was running out of memory and I was investigating the > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > > > > > which aren't cleaned by the GC. > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data > > > > > (see > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > completely and is kept in memory? strvalConnected is also set to > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > > > > > ideas what might cause this? > > > > > > > > > > Thanks, > > > > > Thibaut ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [July 20, 2011, 8:41pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/7 "2011-07-20T20:41:14Z")
Ok, On one node I also observerved over 150 instances of NioClientSocketChannel, but only at 16 MB each. So getting netty to reduce the channel buffer afterwards won't work... I never have more than 10 threads querying in parallel, so I'm a little suprised that there are more than 150 instances. Is this related to the scroll window of 5 min? Or maybe to some errors/timeouts on the search site which won't cause elasticsearch to close the channel? Thanks, Thibaut 150 instances of "org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel", loaded by "sun.misc.Launcher$AppClassLoader @ 0x77cc1e800" occupy 732,664,720 (48.63%) bytes. Biggest instances: org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d23f40 - 16,778,904 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d2c580 - 16,778,904 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d22c08 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d271a8 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d299a8 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d42d48 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797e0c3e0 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797ea3570 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f643a8 - 16,778,856 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797cdcaf0 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797cdea28 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797d00a18 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797e0d098 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f42f48 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f4bad0 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f55cb0 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f56988 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f5a9a0 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f5bf30 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f63d20 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f665c8 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f69b60 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f6b7b0 - 16,778,808 (1.11%) bytes. org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel @ 0x797f6d0e8 - 16,778,808 (1.11%) bytes. On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Heya, > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > Hi, > > > > I'm scrolling over a scan: > > > > SearchRequestBuilder srb = > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > I only requested the keys, but just saw that I was requesting 10000 > > hits per shard. I have over \> 50 shards, but not all of them return > > results. But I assume I got about 200000 results per scroll. I limited > > that. > > > > Changing the cached Pool to a thread pool which releases the threads > > after a few seconds should fix this? > > No, thats not how netty works..., started a discussion with Trustin to see > if we can improve on it. > > > Thanks, > > Thibaut > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Are you sending large messages around? For example, large bulk requests, > > > asking for a very large number of hits in one request? > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > Hi, > > > > > > > > the data is being kept in the buffer array of the > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > at > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > Thanks, > > > > Thibaut > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Our application was running out of memory and I was investigating > > > > > > the > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data > > > > > > (see > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > completely and is kept in memory? strvalConnected is also set to > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > > > > > > ideas what might cause this? > > > > > > > > > > > > Thanks, > > > > > > Thibaut ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 20, 2011, 8:55pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/8 "2011-07-20T20:55:46Z")
How many clients are you indexing with? Do you have a single Node instance? How many servers in the cluster? On Wed, Jul 20, 2011 at 11:41 PM, Thibaut Britz \< [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote: > Ok, > > On one node I also observerved over 150 instances of > NioClientSocketChannel, but only at 16 MB each. So getting netty to > reduce the channel buffer afterwards won't work... > > I never have more than 10 threads querying in parallel, so I'm a > little suprised that there are more than 150 instances. > > Is this related to the scroll window of 5 min? Or maybe to some > errors/timeouts on the search site which won't cause elasticsearch to > close the channel? > > Thanks, > Thibaut > > 150 instances of > "org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel", > loaded by "sun.misc.Launcher$AppClassLoader @ 0x77cc1e800" occupy > 732,664,720 (48.63%) bytes. > > Biggest instances: > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d23f40 - 16,778,904 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d2c580 - 16,778,904 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d22c08 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d271a8 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d299a8 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d42d48 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797e0c3e0 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797ea3570 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f643a8 - 16,778,856 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797cdcaf0 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797cdea28 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797d00a18 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797e0d098 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f42f48 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f4bad0 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f55cb0 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f56988 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f5a9a0 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f5bf30 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f63d20 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f665c8 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f69b60 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f6b7b0 - 16,778,808 (1.11%) bytes. > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > @ 0x797f6d0e8 - 16,778,808 (1.11%) bytes. > > On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Heya, > > > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > Hi, > > > > > > I'm scrolling over a scan: > > > > > > SearchRequestBuilder srb = > > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > > > I only requested the keys, but just saw that I was requesting 10000 > > > hits per shard. I have over \> 50 shards, but not all of them return > > > results. But I assume I got about 200000 results per scroll. I limited > > > that. > > > > > > Changing the cached Pool to a thread pool which releases the threads > > > after a few seconds should fix this? > > > > No, thats not how netty works..., started a discussion with Trustin to > > see > > if we can improve on it. > > > > > Thanks, > > > Thibaut > > > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Are you sending large messages around? For example, large bulk > > > > requests, > > > > asking for a very large number of hits in one request? > > > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > the data is being kept in the buffer array of the > > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > > > at > > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > > > Thanks, > > > > > Thibaut > > > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Our application was running out of memory and I was investigating > > > > > > > the > > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel > > > > > > > instances, > > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data > > > > > > > (see > > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > > completely and is kept in memory? strvalConnected is also set to > > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. > > > > > > > Any > > > > > > > ideas what might cause this? > > > > > > > > > > > > > > Thanks, > > > > > > > Thibaut ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [July 20, 2011, 9:14pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/9 "2011-07-20T21:14:22Z")
All indexing is done through rivers (about 10 in total), this is node is only querying, not doing any indexing. 40 search servers and I connect to a pool of 15 search servers. About 100 shards, with one replica each. It's the same setup as Mechel Conrad described. Thanks, Thibaut On Wed, Jul 20, 2011 at 10:55 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > How many clients are you indexing with? Do you have a single Node instance? > How many servers in the cluster? > > On Wed, Jul 20, 2011 at 11:41 PM, Thibaut Britz > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > Ok, > > > > On one node I also observerved over 150 instances of > > NioClientSocketChannel, but only at 16 MB each. So getting netty to > > reduce the channel buffer afterwards won't work... > > > > I never have more than 10 threads querying in parallel, so I'm a > > little suprised that there are more than 150 instances. > > > > Is this related to the scroll window of 5 min? Or maybe to some > > errors/timeouts on the search site which won't cause elasticsearch to > > close the channel? > > > > Thanks, > > Thibaut > > > > 150 instances of > > > > "org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel", > > loaded by "sun.misc.Launcher$AppClassLoader @ 0x77cc1e800" occupy > > 732,664,720 (48.63%) bytes. > > > > Biggest instances: > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d23f40 - 16,778,904 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d2c580 - 16,778,904 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d22c08 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d271a8 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d299a8 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d42d48 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797e0c3e0 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797ea3570 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f643a8 - 16,778,856 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797cdcaf0 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797cdea28 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797d00a18 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797e0d098 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f42f48 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f4bad0 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f55cb0 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f56988 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f5a9a0 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f5bf30 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f63d20 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f665c8 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f69b60 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f6b7b0 - 16,778,808 (1.11%) bytes. > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > @ 0x797f6d0e8 - 16,778,808 (1.11%) bytes. > > > > On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Heya, > > > > > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > Hi, > > > > > > > > I'm scrolling over a scan: > > > > > > > > SearchRequestBuilder srb = > > > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > > > > > I only requested the keys, but just saw that I was requesting 10000 > > > > hits per shard. I have over \> 50 shards, but not all of them return > > > > results. But I assume I got about 200000 results per scroll. I limited > > > > that. > > > > > > > > Changing the cached Pool to a thread pool which releases the threads > > > > after a few seconds should fix this? > > > > > > No, thats not how netty works..., started a discussion with Trustin to > > > see > > > if we can improve on it. > > > > > > > Thanks, > > > > Thibaut > > > > > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > Are you sending large messages around? For example, large bulk > > > > > requests, > > > > > asking for a very large number of hits in one request? > > > > > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > the data is being kept in the buffer array of the > > > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > > > > > at > > > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > > > > > > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > > > > > Thanks, > > > > > > Thibaut > > > > > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Our application was running out of memory and I was investigating > > > > > > > > the > > > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel > > > > > > > > instances, > > > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data > > > > > > > > (see > > > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > > > completely and is kept in memory? strvalConnected is also set to > > > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. > > > > > > > > Any > > > > > > > > ideas what might cause this? > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Thibaut ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 20, 2011, 9:45pm UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/10 "2011-07-20T21:45:21Z")
What do you mean by 15 search servers? Are those basically client nodes? Do you have one Node instance per "search server"? Internally, nodes creates several connections between them in order to improve on network throughput, you can configure that with other values (I don't think its documented, I'll document that)... On Thu, Jul 21, 2011 at 12:14 AM, Thibaut Britz \< [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote: > All indexing is done through rivers (about 10 in total), this is node > is only querying, not doing any indexing. > > 40 search servers and I connect to a pool of 15 search servers. > > About 100 shards, with one replica each. > > It's the same setup as Mechel Conrad described. > > Thanks, > Thibaut > > On Wed, Jul 20, 2011 at 10:55 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > How many clients are you indexing with? Do you have a single Node > > instance? > > How many servers in the cluster? > > > > On Wed, Jul 20, 2011 at 11:41 PM, Thibaut Britz > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > Ok, > > > > > > On one node I also observerved over 150 instances of > > > NioClientSocketChannel, but only at 16 MB each. So getting netty to > > > reduce the channel buffer afterwards won't work... > > > > > > I never have more than 10 threads querying in parallel, so I'm a > > > little suprised that there are more than 150 instances. > > > > > > Is this related to the scroll window of 5 min? Or maybe to some > > > errors/timeouts on the search site which won't cause elasticsearch to > > > close the channel? > > > > > > Thanks, > > > Thibaut > > > > > > 150 instances of > > "org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel", > > > > loaded by "sun.misc.Launcher$AppClassLoader @ 0x77cc1e800" occupy > > > 732,664,720 (48.63%) bytes. > > > > > > Biggest instances: > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d23f40 - 16,778,904 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d2c580 - 16,778,904 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d22c08 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d271a8 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d299a8 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d42d48 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797e0c3e0 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797ea3570 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f643a8 - 16,778,856 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797cdcaf0 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797cdea28 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797d00a18 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797e0d098 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f42f48 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f4bad0 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f55cb0 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f56988 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f5a9a0 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f5bf30 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f63d20 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f665c8 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f69b60 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f6b7b0 - 16,778,808 (1.11%) bytes. > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > @ 0x797f6d0e8 - 16,778,808 (1.11%) bytes. > > > > > > On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Heya, > > > > > > > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm scrolling over a scan: > > > > > > > > > > SearchRequestBuilder srb = > > > > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > > > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > > > > > > > I only requested the keys, but just saw that I was requesting 10000 > > > > > hits per shard. I have over \> 50 shards, but not all of them return > > > > > results. But I assume I got about 200000 results per scroll. I > > > > > limited > > > > > that. > > > > > > > > > > Changing the cached Pool to a thread pool which releases the threads > > > > > after a few seconds should fix this? > > > > > > > > No, thats not how netty works..., started a discussion with Trustin to > > > > see > > > > if we can improve on it. > > > > > > > > > Thanks, > > > > > Thibaut > > > > > > > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > Are you sending large messages around? For example, large bulk > > > > > > requests, > > > > > > asking for a very large number of hits in one request? > > > > > > > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > the data is being kept in the buffer array of the > > > > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > > > > > > > at > > > > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > > > > > > > Thanks, > > > > > > > Thibaut > > > > > > > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > Our application was running out of memory and I was > > > > > > > > > investigating > > > > > > > > > the > > > > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel > > > > > > > > > instances, > > > > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced > > > > > > > > > data > > > > > > > > > (see > > > > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > > > > completely and is kept in memory? strvalConnected is also set > > > > > > > > > to > > > > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch > > > > > > > > > applied. > > > > > > > > > Any > > > > > > > > > ideas what might cause this? > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Thibaut ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [July 21, 2011, 8:27am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/11 "2011-07-21T08:27:29Z")
There are 40 dedicated servers running elasticsearch only. I connect to a pool to 15 of these dedicated search servers in my application through: TransportClient c = new TransportClient(ImmutableSettings.settingsBuilder().put("cluster.name", clustername)); for (String h : hosts) { c.addTransportAddress(new InetSocketTransportAddress(h, port)); } and execute a search. This probably explains why I have 150 instances. Still, releasing the buffer after the data has been transfered makes perfectly sense in this case. Thibaut On Wed, Jul 20, 2011 at 11:45 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > What do you mean by 15 search servers? Are those basically client nodes? Do > you have one Node instance per "search server"? Internally, nodes creates > several connections between them in order to improve on network throughput, > you can configure that with other values (I don't think its documented, I'll > document that)... > > On Thu, Jul 21, 2011 at 12:14 AM, Thibaut Britz > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > All indexing is done through rivers (about 10 in total), this is node > > is only querying, not doing any indexing. > > > > 40 search servers and I connect to a pool of 15 search servers. > > > > About 100 shards, with one replica each. > > > > It's the same setup as Mechel Conrad described. > > > > Thanks, > > Thibaut > > > > On Wed, Jul 20, 2011 at 10:55 PM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > How many clients are you indexing with? Do you have a single Node > > > instance? > > > How many servers in the cluster? > > > > > > On Wed, Jul 20, 2011 at 11:41 PM, Thibaut Britz > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > Ok, > > > > > > > > On one node I also observerved over 150 instances of > > > > NioClientSocketChannel, but only at 16 MB each. So getting netty to > > > > reduce the channel buffer afterwards won't work... > > > > > > > > I never have more than 10 threads querying in parallel, so I'm a > > > > little suprised that there are more than 150 instances. > > > > > > > > Is this related to the scroll window of 5 min? Or maybe to some > > > > errors/timeouts on the search site which won't cause elasticsearch to > > > > close the channel? > > > > > > > > Thanks, > > > > Thibaut > > > > > > > > 150 instances of > > > > > > > > "org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel", > > > > loaded by "sun.misc.Launcher$AppClassLoader @ 0x77cc1e800" occupy > > > > 732,664,720 (48.63%) bytes. > > > > > > > > Biggest instances: > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d23f40 - 16,778,904 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d2c580 - 16,778,904 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d22c08 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d271a8 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d299a8 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d42d48 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797e0c3e0 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797ea3570 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f643a8 - 16,778,856 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797cdcaf0 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797cdea28 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797d00a18 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797e0d098 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f42f48 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f4bad0 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f55cb0 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f56988 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f5a9a0 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f5bf30 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f63d20 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f665c8 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f69b60 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f6b7b0 - 16,778,808 (1.11%) bytes. > > > > > > > > org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketChannel > > > > @ 0x797f6d0e8 - 16,778,808 (1.11%) bytes. > > > > > > > > On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > Heya, > > > > > > > > > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm scrolling over a scan: > > > > > > > > > > > > SearchRequestBuilder srb = > > > > > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > > > > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > > > > > > > > > I only requested the keys, but just saw that I was requesting 10000 > > > > > > hits per shard. I have over \> 50 shards, but not all of them return > > > > > > results. But I assume I got about 200000 results per scroll. I > > > > > > limited > > > > > > that. > > > > > > > > > > > > Changing the cached Pool to a thread pool which releases the threads > > > > > > after a few seconds should fix this? > > > > > > > > > > No, thats not how netty works..., started a discussion with Trustin > > > > > to > > > > > see > > > > > if we can improve on it. > > > > > > > > > > > Thanks, > > > > > > Thibaut > > > > > > > > > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > Are you sending large messages around? For example, large bulk > > > > > > > requests, > > > > > > > asking for a very large number of hits in one request? > > > > > > > > > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > the data is being kept in the buffer array of the > > > > > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > > > > > > > > > at > > > > > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > > > > > > > > > > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Thibaut > > > > > > > > > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > Our application was running out of memory and I was > > > > > > > > > > investigating > > > > > > > > > > the > > > > > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel > > > > > > > > > > instances, > > > > > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced > > > > > > > > > > data > > > > > > > > > > (see > > > > > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > > > > > completely and is kept in memory? strvalConnected is also set > > > > > > > > > > to > > > > > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch > > > > > > > > > > applied. > > > > > > > > > > Any > > > > > > > > > > ideas what might cause this? > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Thibaut ---
**Author:** ![Thibaut\_Britz](https://avatars.discourse-cdn.com/v4/letter/t/a6a055/32.png) [@Thibaut\_Britz](https://discuss.elastic.co/u/Thibaut_Britz)\ **Post date:** [August 16, 2011, 8:57am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/12 "2011-08-16T08:57:43Z")
Any update on this? Thanks, Thibaut On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Heya, > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > Hi, > > > > I'm scrolling over a scan: > > > > SearchRequestBuilder srb = > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > I only requested the keys, but just saw that I was requesting 10000 > > hits per shard. I have over \> 50 shards, but not all of them return > > results. But I assume I got about 200000 results per scroll. I limited > > that. > > > > Changing the cached Pool to a thread pool which releases the threads > > after a few seconds should fix this? > > No, thats not how netty works..., started a discussion with Trustin to see > if we can improve on it. > > > Thanks, > > Thibaut > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > Are you sending large messages around? For example, large bulk requests, > > > asking for a very large number of hits in one request? > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > Hi, > > > > > > > > the data is being kept in the buffer array of the > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > at > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > Thanks, > > > > Thibaut > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > Our application was running out of memory and I was investigating > > > > > > the > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel instances, > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data > > > > > > (see > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > completely and is kept in memory? strvalConnected is also set to > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. Any > > > > > > ideas what might cause this? > > > > > > > > > > > > Thanks, > > > > > > Thibaut ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 16, 2011, 11:05am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/13 "2011-08-16T11:05:27Z")
No, no updates... On Tue, Aug 16, 2011 at 11:57 AM, Thibaut Britz \< [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com)\> wrote: > Any update on this? > > Thanks, > Thibaut > > On Tue, Jul 19, 2011 at 8:58 PM, Shay Banon > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > Heya, > > > > On Tue, Jul 19, 2011 at 1:19 PM, Thibaut Britz > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > Hi, > > > > > > I'm scrolling over a scan: > > > > > > SearchRequestBuilder srb = > > > searchclient.getClient().prepareSearch(indexnames.toArray(new > > > String[0])).setSearchType(SearchType.SCAN).setScroll("10m"); > > > > > > I only requested the keys, but just saw that I was requesting 10000 > > > hits per shard. I have over \> 50 shards, but not all of them return > > > results. But I assume I got about 200000 results per scroll. I limited > > > that. > > > > > > Changing the cached Pool to a thread pool which releases the threads > > > after a few seconds should fix this? > > > > No, thats not how netty works..., started a discussion with Trustin to > > see > > if we can improve on it. > > > > > Thanks, > > > Thibaut > > > > > > On Tue, Jul 19, 2011 at 12:01 AM, Shay Banon > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > Are you sending large messages around? For example, large bulk > > > > requests, > > > > asking for a very large number of hits in one request? > > > > > > > > On Mon, Jul 18, 2011 at 11:40 PM, Thibaut Britz > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > the data is being kept in the buffer array of the > > > > > BigEndianHeapChannelBuffer instances in jetty (see screenshot). > > > > > > > > > > at > > > > > org.elasticsearch.common.netty.buffer.BigEndianHeapChannelBuffer > > > > > org.elasticsearch.common.netty.buffer.DynamicChannelBuffer > > > > > org.elasticsearch.transport.netty.SizeHeaderFrameDecoder > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext > > > > > > org.elasticsearch.common.netty.channel.DefaultChannelPipeline > > > > > > > > > > Thanks, > > > > > Thibaut > > > > > > > > > > On Mon, Jul 18, 2011 at 6:57 PM, Shay Banon > > > > > [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > > > > > > > > > Can you dig down a bit more and see where its being used? > > > > > > > > > > > > On Mon, Jul 18, 2011 at 1:24 PM, Thibaut Britz > > > > > > [thibaut.britz@trendiction.com](mailto:thibaut.britz@trendiction.com) wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > Our application was running out of memory and I was investigating > > > > > > > the > > > > > > > issue. It tourned out that elasticsearch was keeping over 600 > > > > > > > Megabytes (4 of them \> 100 MB) of NioClientSocketChannel > > > > > > > instances, > > > > > > > which aren't cleaned by the GC. > > > > > > > > > > > > > > I analyzed the heap in MAT but coultdn't find any referenced data > > > > > > > (see > > > > > > > screenshot).Is this due to a NIO buffer which isn't read out > > > > > > > completely and is kept in memory? strvalConnected is also set to > > > > > > > false, thus the memory should get reclaimed. > > > > > > > > > > > > > > I'm using 0.16-4 Snapshot with the gc native leak patch applied. > > > > > > > Any > > > > > > > ideas what might cause this? > > > > > > > > > > > > > > Thanks, > > > > > > > Thibaut ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:56am UTC](https://discuss.elastic.co/t/memory-leak-in-nioclientsocketchannel/4881/14 "2017-07-06T03:56:51Z")