# HDFS Gateway - SendRequestTransportException **URL:** \ **Category:** Elasticsearch\ **Created:** [June 8, 2010, 3:54pm UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002 "2010-06-08T15:54:53Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Zaharije](https://avatars.discourse-cdn.com/v4/letter/z/ebca7d/32.png) [@Zaharije](https://discuss.elastic.co/u/Zaharije)\ **Post date:** [June 8, 2010, 3:54pm UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/1 "2010-06-08T15:54:53Z")
Hi i'm writing small multi threaded app to index some random data (each thread has it's own client). With default gateway it works fine. I indexed 10M documents in about 90min. ES cluster is small one - 8 nodes with 4 cores + 4GB memory. When I installed hdfs gateway, i got strange behavior - sometimes i'm getting following exception on client side (im using TransportClient): Exception in thread "Thread-7" org.elasticsearch.transport.SendRequestTransportException: [Wonder Man] [inet[/172.17.12.1:9300]][indices/index/shard/index] at org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: 58) at org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: 34) at java.lang.Throwable.(Throwable.java:218) at java.lang.Exception.(Exception.java:59) at java.lang.RuntimeException.(RuntimeException.java:61) at org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: 46) at org.elasticsearch.transport.TransportException.(TransportException.java: 34) at org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: 39) at org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: 30) at org.elasticsearch.transport.TransportService $2.run(TransportService.java:197) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:637) Caused by: java.lang.NullPointerException at org.elasticsearch.util.io.stream.BytesStreamOutput $Cached.cachedHandles(BytesStreamOutput.java:59) at org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: 385) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: 183) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: 171) at org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: 70) at org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: 65) at org.elasticsearch.client.transport.support.InternalTransportClient $1.doWithNode(InternalTransportClient.java:125) at org.elasticsearch.client.transport.support.InternalTransportClient $1.doWithNode(InternalTransportClient.java:123) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: 144) at org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: 123) at org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: 223) at com.test.App$IndexThread.index(App.java:115) at com.test.App$IndexThread.run(App.java:34) HDFS is also clustered across 13 nodes, and i do not see any errors in namenode. Any ideas? Regards Zaharije ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 9, 2010, 2:40pm UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/2 "2010-06-09T14:40:47Z")
Do you get this exception consistently? Strange one... -shay.banon On Tue, Jun 8, 2010 at 6:54 PM, Zaharije [pasalic.zaharije@gmail.com](mailto:pasalic.zaharije@gmail.com) wrote: > Hi > > i'm writing small multi threaded app to index some random data (each > thread has it's own client). With default gateway it works fine. I > indexed 10M documents in about 90min. > > ES cluster is small one - 8 nodes with 4 cores + 4GB memory. > > When I installed hdfs gateway, i got strange behavior - sometimes i'm > getting following exception on client side (im using TransportClient): > > Exception in thread "Thread-7" > org.elasticsearch.transport.SendRequestTransportException: [Wonder Man] > [inet[/172.17.12.1:9300]][indices/index/shard/index] > at > > org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: > 58) > at > > org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: > 34) > at java.lang.Throwable.(Throwable.java:218) > at java.lang.Exception.(Exception.java:59) > at java.lang.RuntimeException.(RuntimeException.java:61) > at > > org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: > 46) > at > > org.elasticsearch.transport.TransportException.(TransportException.java: > 34) > at > > org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: > 39) > at > > org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: > 30) > at org.elasticsearch.transport.TransportService > $2.run(TransportService.java:197) > at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:886) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:637) > Caused by: java.lang.NullPointerException > at org.elasticsearch.util.io.stream.BytesStreamOutput > $Cached.cachedHandles(BytesStreamOutput.java:59) > at > > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: > 385) > at > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > 183) > at > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > 171) > at > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > 70) > at > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > 65) > at > org.elasticsearch.client.transport.support.InternalTransportClient > $1.doWithNode(InternalTransportClient.java:125) > at > org.elasticsearch.client.transport.support.InternalTransportClient > $1.doWithNode(InternalTransportClient.java:123) > at > > org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: > 144) > at > > org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: > 123) > at > > org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: > 223) > at com.test.App$IndexThread.index(App.java:115) > at com.test.App$IndexThread.run(App.java:34) > > HDFS is also clustered across 13 nodes, and i do not see any errors in > namenode. > Any ideas? > > Regards > Zaharije ---
**Author:** ![Zaharije](https://avatars.discourse-cdn.com/v4/letter/z/ebca7d/32.png) [@Zaharije](https://discuss.elastic.co/u/Zaharije)\ **Post date:** [June 11, 2010, 9:55pm UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/3 "2010-06-11T21:55:52Z")
Yes. It seems that this is more HDFS related problem. After few runs i'm getting timeout exceptions from hdfs, but when i lower my thread/process count (creating index) than it looks fine. In general, is it good idea to store lucene indexes into hdfs? I thought that hdfs has really bad performance for r/w of the small files ? Regards Zaharije On Wed, Jun 9, 2010 at 4:40 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > Do you get this exception consistently? Strange one... > -shay.banon > > On Tue, Jun 8, 2010 at 6:54 PM, Zaharije [pasalic.zaharije@gmail.com](mailto:pasalic.zaharije@gmail.com) wrote: > > > Hi > > > > i'm writing small multi threaded app to index some random data (each > > thread has it's own client). With default gateway it works fine. I > > indexed 10M documents in about 90min. > > > > ES cluster is small one - 8 nodes with 4 cores + 4GB memory. > > > > When I installed hdfs gateway, i got strange behavior - sometimes i'm > > getting following exception on client side (im using TransportClient): > > > > Exception in thread "Thread-7" > > org.elasticsearch.transport.SendRequestTransportException: [Wonder Man] > > [inet[/172.17.12.1:9300]][indices/index/shard/index] > > at > > > > org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: > > 58) > > at > > > > org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: > > 34) > > at java.lang.Throwable.(Throwable.java:218) > > at java.lang.Exception.(Exception.java:59) > > at java.lang.RuntimeException.(RuntimeException.java:61) > > at > > > > org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: > > 46) > > at > > > > org.elasticsearch.transport.TransportException.(TransportException.java: > > 34) > > at > > > > org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: > > 39) > > at > > > > org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: > > 30) > > at org.elasticsearch.transport.TransportService > > $2.run(TransportService.java:197) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:637) > > Caused by: java.lang.NullPointerException > > at org.elasticsearch.util.io.stream.BytesStreamOutput > > $Cached.cachedHandles(BytesStreamOutput.java:59) > > at > > > > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: > > 385) > > at > > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > 183) > > at > > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > 171) > > at > > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > 70) > > at > > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > 65) > > at > > org.elasticsearch.client.transport.support.InternalTransportClient > > $1.doWithNode(InternalTransportClient.java:125) > > at > > org.elasticsearch.client.transport.support.InternalTransportClient > > $1.doWithNode(InternalTransportClient.java:123) > > at > > > > org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: > > 144) > > at > > > > org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: > > 123) > > at > > > > org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: > > 223) > > at com.test.App$IndexThread.index(App.java:115) > > at com.test.App$IndexThread.run(App.java:34) > > > > HDFS is also clustered across 13 nodes, and i do not see any errors in > > namenode. > > Any ideas? > > > > Regards > > Zaharije ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 12, 2010, 4:00pm UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/4 "2010-06-12T16:00:03Z")
The HDFS support is done in the gateway, the actual indices that elasticsearch works with are stored on the local file system (by default). I talk about the gateway role in elasticsearch here: [http://www.elasticsearch.com/blog/2010/02/16/searchengine\_time\_machine.html](http://www.elasticsearch.com/blog/2010/02/16/searchengine_time_machine.html). -shay.banon On Sat, Jun 12, 2010 at 12:55 AM, Zaharije Pasalic \< [pasalic.zaharije@gmail.com](mailto:pasalic.zaharije@gmail.com)\> wrote: > Yes. > > It seems that this is more HDFS related problem. After few runs i'm > getting timeout exceptions from hdfs, but when i lower my > thread/process count (creating index) than it looks fine. > > In general, is it good idea to store lucene indexes into hdfs? I > thought that hdfs has really bad performance for r/w of the small > files ? > > Regards > Zaharije > > On Wed, Jun 9, 2010 at 4:40 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) > wrote: > > > Do you get this exception consistently? Strange one... > > -shay.banon > > > > On Tue, Jun 8, 2010 at 6:54 PM, Zaharije [pasalic.zaharije@gmail.com](mailto:pasalic.zaharije@gmail.com) > > wrote: > > > > > Hi > > > > > > i'm writing small multi threaded app to index some random data (each > > > thread has it's own client). With default gateway it works fine. I > > > indexed 10M documents in about 90min. > > > > > > ES cluster is small one - 8 nodes with 4 cores + 4GB memory. > > > > > > When I installed hdfs gateway, i got strange behavior - sometimes i'm > > > getting following exception on client side (im using TransportClient): > > > > > > Exception in thread "Thread-7" > > > org.elasticsearch.transport.SendRequestTransportException: [Wonder Man] > > > [inet[/172.17.12.1:9300]][indices/index/shard/index] > > > at > > org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: > > > > 1. at > > org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: > > > > 1. at java.lang.Throwable.(Throwable.java:218) > > > at java.lang.Exception.(Exception.java:59) > > > at java.lang.RuntimeException.(RuntimeException.java:61) > > > at > > org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: > > > > 1. at > > org.elasticsearch.transport.TransportException.(TransportException.java: > > > > 1. at > > org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: > > > > 1. at > > org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: > > > > 1. at org.elasticsearch.transport.TransportService > > > $2.run(TransportService.java:197) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:908) > > > at java.lang.Thread.run(Thread.java:637) > > > Caused by: java.lang.NullPointerException > > > at org.elasticsearch.util.io.stream.BytesStreamOutput > > > $Cached.cachedHandles(BytesStreamOutput.java:59) > > > at > > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: > > > > 1. at > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > 1. at > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > 1. at > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > 1. at > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > 1. at > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > $1.doWithNode(InternalTransportClient.java:125) > > > at > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > $1.doWithNode(InternalTransportClient.java:123) > > > at > > org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: > > > > 1. at > > org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: > > > > 1. at > > org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: > > > > 1. at com.test.App$IndexThread.index(App.java:115) > > > at com.test.App$IndexThread.run(App.java:34) > > > > > > HDFS is also clustered across 13 nodes, and i do not see any errors in > > > namenode. > > > Any ideas? > > > > > > Regards > > > Zaharije ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 14, 2010, 2:01am UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/5 "2010-06-14T02:01:27Z")
Bok Zaharije, For what it's worth, and just to state it explicitly: HDFS is great for storing indices because it automatically creates their replicas, so one doesn't have to make backups. But, yes, you shouldn't search indices stored in HDFS. ## Otis Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/) On Jun 12, 12:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > The HDFS support is done in the gateway, the actual indices that > elasticsearch works with are stored on the local file system (by default). I > talk about the gateway role in elasticsearch here:[http://www.elasticsearch.com/blog/2010/02/16/searchengine\_time\_machin](http://www.elasticsearch.com/blog/2010/02/16/searchengine_time_machin).... > > -shay.banon > > On Sat, Jun 12, 2010 at 12:55 AM, Zaharije Pasalic \< > > [pasalic.zahar...@gmail.com](mailto:pasalic.zahar...@gmail.com)\> wrote: > > > Yes. > > > It seems that this is more HDFS related problem. After few runs i'm > > getting timeout exceptions from hdfs, but when i lower my > > thread/process count (creating index) than it looks fine. > > > In general, is it good idea to store lucene indexes into hdfs? I > > thought that hdfs has really bad performance for r/w of the small > > files ? > > > Regards > > Zaharije > > > On Wed, Jun 9, 2010 at 4:40 PM, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) > > wrote: > > > > > Do you get this exception consistently? Strange one... > > > -shay.banon > > > > On Tue, Jun 8, 2010 at 6:54 PM, Zaharije [pasalic.zahar...@gmail.com](mailto:pasalic.zahar...@gmail.com) > > > wrote: > > > > > Hi > > > > > i'm writing small multi threaded app to index some random data (each > > > > thread has it's own client). With default gateway it works fine. I > > > > indexed 10M documents in about 90min. > > > > > ES cluster is small one - 8 nodes with 4 cores + 4GB memory. > > > > > When I installed hdfs gateway, i got strange behavior - sometimes i'm > > > > getting following exception on client side (im using TransportClient): > > > > > Exception in thread "Thread-7" > > > > org.elasticsearch.transport.SendRequestTransportException: [Wonder Man] > > > > [inet[/172.17.12.1:9300]][indices/index/shard/index] > > > > at > > > org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: > > > > > > 1. at > > > org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: > > > > > > 1. at java.lang.Throwable.(Throwable.java:218) > > > > at java.lang.Exception.(Exception.java:59) > > > > at java.lang.RuntimeException.(RuntimeException.java:61) > > > > at > > > org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: > > > > > > 1. at > > > org.elasticsearch.transport.TransportException.(TransportException.java: > > > > > > 1. at > > > org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: > > > > > > 1. at > > > org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: > > > > > > 1. at org.elasticsearch.transport.TransportService > > > > $2.run(TransportService.java:197) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:637) > > > > Caused by: java.lang.NullPointerException > > > > at org.elasticsearch.util.io.stream.BytesStreamOutput > > > > $Cached.cachedHandles(BytesStreamOutput.java:59) > > > > at > > > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: > > > > > > 1. at > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > > > 1. at > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > > > 1. at > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > > > 1. at > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > > > 1. at > > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > > $1.doWithNode(InternalTransportClient.java:125) > > > > at > > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > > $1.doWithNode(InternalTransportClient.java:123) > > > > at > > > org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: > > > > > > 1. at > > > org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: > > > > > > 1. at > > > org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: > > > > > > 1. at com.test.App$IndexThread.index(App.java:115) > > > > at com.test.App$IndexThread.run(App.java:34) > > > > > HDFS is also clustered across 13 nodes, and i do not see any errors in > > > > namenode. > > > > Any ideas? > > > > > Regards > > > > Zaharije ---
**Author:** ![Zaharije](https://avatars.discourse-cdn.com/v4/letter/z/ebca7d/32.png) [@Zaharije](https://discuss.elastic.co/u/Zaharije)\ **Post date:** [June 16, 2010, 8:52am UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/6 "2010-06-16T08:52:16Z")
So if I understand it: Gateway is used for long time persistence storage and i can control it via gateway modules (like hadoop, fs, cloud). And index gateway is user for per node (is it in fact per shard?) level. So if i had enough memory i can user let say hdfs for long term storage, and memory for shard level? thx Zaharije On Mon, Jun 14, 2010 at 4:01 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > Bok Zaharije, > > For what it's worth, and just to state it explicitly: HDFS is great > for storing indices because it automatically creates their replicas, > so one doesn't have to make backups. But, yes, you shouldn't search > indices stored in HDFS. > > ## Otis > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch > Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/) > > On Jun 12, 12:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > The HDFS support is done in the gateway, the actual indices that > > elasticsearch works with are stored on the local file system (by default). I > > talk about the gateway role in elasticsearch here:[http://www.elasticsearch.com/blog/2010/02/16/searchengine\_time\_machin](http://www.elasticsearch.com/blog/2010/02/16/searchengine_time_machin).... > > > > -shay.banon > > > > On Sat, Jun 12, 2010 at 12:55 AM, Zaharije Pasalic \< > > > > [pasalic.zahar...@gmail.com](mailto:pasalic.zahar...@gmail.com)\> wrote: > > > > > Yes. > > > > > It seems that this is more HDFS related problem. After few runs i'm > > > getting timeout exceptions from hdfs, but when i lower my > > > thread/process count (creating index) than it looks fine. > > > > > In general, is it good idea to store lucene indexes into hdfs? I > > > thought that hdfs has really bad performance for r/w of the small > > > files ? > > > > > Regards > > > Zaharije > > > > > On Wed, Jun 9, 2010 at 4:40 PM, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) > > > wrote: > > > > > > > Do you get this exception consistently? Strange one... > > > > -shay.banon > > > > > > On Tue, Jun 8, 2010 at 6:54 PM, Zaharije [pasalic.zahar...@gmail.com](mailto:pasalic.zahar...@gmail.com) > > > > wrote: > > > > > > > Hi > > > > > > > i'm writing small multi threaded app to index some random data (each > > > > > thread has it's own client). With default gateway it works fine. I > > > > > indexed 10M documents in about 90min. > > > > > > > ES cluster is small one - 8 nodes with 4 cores + 4GB memory. > > > > > > > When I installed hdfs gateway, i got strange behavior - sometimes i'm > > > > > getting following exception on client side (im using TransportClient): > > > > > > > Exception in thread "Thread-7" > > > > > org.elasticsearch.transport.SendRequestTransportException: [Wonder Man] > > > > > [inet[/172.17.12.1:9300]][indices/index/shard/index] > > > > > at > > > > > org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: > > > > > > > > 1. at > > > > > org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: > > > > > > > > 1. at java.lang.Throwable.(Throwable.java:218) > > > > > at java.lang.Exception.(Exception.java:59) > > > > > at java.lang.RuntimeException.(RuntimeException.java:61) > > > > > at > > > > > org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: > > > > > > > > 1. at > > > > > org.elasticsearch.transport.TransportException.(TransportException.java: > > > > > > > > 1. at > > > > > org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: > > > > > > > > 1. at > > > > > org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: > > > > > > > > 1. at org.elasticsearch.transport.TransportService > > > > > $2.run(TransportService.java:197) > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > at java.lang.Thread.run(Thread.java:637) > > > > > Caused by: java.lang.NullPointerException > > > > > at org.elasticsearch.util.io.stream.BytesStreamOutput > > > > > $Cached.cachedHandles(BytesStreamOutput.java:59) > > > > > at > > > > > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: > > > > > > > > 1. at > > > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > > > > > 1. at > > > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > > > > > 1. at > > > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > > > > > 1. at > > > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > > > > > 1. at > > > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > > > $1.doWithNode(InternalTransportClient.java:125) > > > > > at > > > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > > > $1.doWithNode(InternalTransportClient.java:123) > > > > > at > > > > > org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: > > > > > > > > 1. at > > > > > org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: > > > > > > > > 1. at > > > > > org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: > > > > > > > > 1. at com.test.App$IndexThread.index(App.java:115) > > > > > at com.test.App$IndexThread.run(App.java:34) > > > > > > > HDFS is also clustered across 13 nodes, and i do not see any errors in > > > > > namenode. > > > > > Any ideas? > > > > > > > Regards > > > > > Zaharije ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 16, 2010, 2:52pm UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/7 "2010-06-16T14:52:55Z")
Gateway is gateway, and its used for long term storage. The index gateway is simply the module responsible for persisting the index data to the gateway. Its there mainly so you can configure, for example, global file system gateway, and disable it for a specific index. Each index (as you correctly said, each shard), stores its state "locally", this is the store module, and it can be either file system, memory, or a combination. -shay.banon On Wed, Jun 16, 2010 at 11:52 AM, Zaharije Pasalic \< [pasalic.zaharije@gmail.com](mailto:pasalic.zaharije@gmail.com)\> wrote: > So if I understand it: > > Gateway is used for long time persistence storage and i can control it > via gateway modules (like hadoop, fs, cloud). And index gateway is > user for per node (is it in fact per shard?) level. > > So if i had enough memory i can user let say hdfs for long term > storage, and memory for shard level? > > thx > Zaharije > > On Mon, Jun 14, 2010 at 4:01 AM, Otis [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com) wrote: > > > Bok Zaharije, > > > > For what it's worth, and just to state it explicitly: HDFS is great > > for storing indices because it automatically creates their replicas, > > so one doesn't have to make backups. But, yes, you shouldn't search > > indices stored in HDFS. > > > > ## Otis > > > > Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Nutch > > Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/) > > > > On Jun 12, 12:00 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > The HDFS support is done in the gateway, the actual indices that > > > elasticsearch works with are stored on the local file system (by > > > default). I > > > talk about the gateway role in elasticsearch here: > > > [http://www.elasticsearch.com/blog/2010/02/16/searchengine\_time\_machin](http://www.elasticsearch.com/blog/2010/02/16/searchengine_time_machin).... > > > > > > -shay.banon > > > > > > On Sat, Jun 12, 2010 at 12:55 AM, Zaharije Pasalic \< > > > > > > [pasalic.zahar...@gmail.com](mailto:pasalic.zahar...@gmail.com)\> wrote: > > > > > > > Yes. > > > > > > > It seems that this is more HDFS related problem. After few runs i'm > > > > getting timeout exceptions from hdfs, but when i lower my > > > > thread/process count (creating index) than it looks fine. > > > > > > > In general, is it good idea to store lucene indexes into hdfs? I > > > > thought that hdfs has really bad performance for r/w of the small > > > > files ? > > > > > > > Regards > > > > Zaharije > > > > > > > On Wed, Jun 9, 2010 at 4:40 PM, Shay Banon \< > > > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\> > > > > wrote: > > > > > > > > > Do you get this exception consistently? Strange one... > > > > > -shay.banon > > > > > > > > On Tue, Jun 8, 2010 at 6:54 PM, Zaharije \< > > > > > [pasalic.zahar...@gmail.com](mailto:pasalic.zahar...@gmail.com)\> > > > > > wrote: > > > > > > > > > Hi > > > > > > > > > i'm writing small multi threaded app to index some random data > > > > > > (each > > > > > > thread has it's own client). With default gateway it works fine. I > > > > > > indexed 10M documents in about 90min. > > > > > > > > > ES cluster is small one - 8 nodes with 4 cores + 4GB memory. > > > > > > > > > When I installed hdfs gateway, i got strange behavior - sometimes > > > > > > i'm > > > > > > getting following exception on client side (im using > > > > > > TransportClient): > > > > > > > > > Exception in thread "Thread-7" > > > > > > org.elasticsearch.transport.SendRequestTransportException: [Wonder > > > > > > Man] > > > > > > [inet[/172.17.12.1:9300]][indices/index/shard/index] > > > > > > at > > > > > > > > > org.elasticsearch.transport.RemoteTransportException.fillStack(RemoteTransportException.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.transport.SendRequestTransportException.fillInStackTrace(SendRequestTransportException.java: > > > > > > > 1. at java.lang.Throwable.(Throwable.java:218) > > > > > > at java.lang.Exception.(Exception.java:59) > > > > > > at > > > > > > java.lang.RuntimeException.(RuntimeException.java:61) > > > > > > at > > > > > > > > > org.elasticsearch.ElasticSearchException.(ElasticSearchException.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.transport.TransportException.(TransportException.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.transport.RemoteTransportException.(RemoteTransportException.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.transport.SendRequestTransportException.(SendRequestTransportException.java: > > > > > > > 1. at org.elasticsearch.transport.TransportService > > > > > > $2.run(TransportService.java:197) > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > $Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > at java.util.concurrent.ThreadPoolExecutor > > > > > > $Worker.run(ThreadPoolExecutor.java:908) > > > > > > at java.lang.Thread.run(Thread.java:637) > > > > > > Caused by: java.lang.NullPointerException > > > > > > at org.elasticsearch.util.io.stream.BytesStreamOutput > > > > > > $Cached.cachedHandles(BytesStreamOutput.java:59) > > > > > > at > > > > > > > > > org.elasticsearch.transport.netty.NettyTransport.sendRequest(NettyTransport.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.transport.TransportService.sendRequest(TransportService.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.client.transport.action.support.BaseClientTransportAction.execute(BaseClientTransportAction.java: > > > > > > > 1. at > > > > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > > > > $1.doWithNode(InternalTransportClient.java:125) > > > > > > at > > > > > > org.elasticsearch.client.transport.support.InternalTransportClient > > > > > > $1.doWithNode(InternalTransportClient.java:123) > > > > > > at > > > > > > > > > org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.client.transport.support.InternalTransportClient.index(InternalTransportClient.java: > > > > > > > 1. at > > > > > > > > > org.elasticsearch.client.transport.TransportClient.index(TransportClient.java: > > > > > > > 1. at com.test.App$IndexThread.index(App.java:115) > > > > > > at com.test.App$IndexThread.run(App.java:34) > > > > > > > > > HDFS is also clustered across 13 nodes, and i do not see any errors > > > > > > in > > > > > > namenode. > > > > > > Any ideas? > > > > > > > > > Regards > > > > > > Zaharije ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:23am UTC](https://discuss.elastic.co/t/hdfs-gateway-sendrequesttransportexception/3002/8 "2017-07-06T04:23:15Z")