# \[Hadoop\] Running elasticsearch-hive in AWS EMR

**URL:** <https://discuss.elastic.co/t/hadoop-running-elasticsearch-hive-in-aws-emr/19579>\
**Category:** Elasticsearch\
**Created:** [September 2, 2014, 10:39pm UTC](https://discuss.elastic.co/t/hadoop-running-elasticsearch-hive-in-aws-emr/19579 "2014-09-02T22:39:32Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jae/32/791_2.png) [@Jae](https://discuss.elastic.co/u/Jae)\
**Post date:** [September 2, 2014, 10:39pm UTC](https://discuss.elastic.co/t/hadoop-running-elasticsearch-hive-in-aws-emr/19579/1 "2014-09-02T22:39:32Z")

</div>

We are running Hadoop and Hive in AWS EMR. Our EMR Hadoop cluster is  
setting Hive table location in S3 and we don't allow any user to write HDFS  
directly. So, when I create a es-hive table, I set up the location in S3,  
but it's still pointing to HDFS and throwing the following exception. I  
appreciate so much for any comments and guidance.

hive\> create external table table\_for\_es (  
\> account\_owner\_id string,  
\> customer\_id string,  
\> update\_operation string,  
\> update\_type string,  
\> updates string,  
\> app\_id string,  
\> ts bigint  
\> )  
\> stored by 'org.elasticsearch.hadoop.hive.EsStorageHandler'  
\> location 's3n://bucket/hive/warehouse/datapipeline.db/tabke\_for\_es'  
\> tblproperties('es.resource'='table/default',  
'es\_nodes'='es\_host:7104');  
OK  
Time taken: 0.423 seconds  
hive\> insert overwrite table table\_es  
\> select other\_properties['account\_owner\_id'],  
other\_properties['customer\_id'], other\_properties['update\_operation'],  
other\_properties['update\_type'], other\_properties['updates'],  
other\_properties['app\_id'], event\_utc\_ms from default.table where  
dateint=20140902 and hour=00;  
Total MapReduce jobs = 1  
Launching Job 1 out of 1  
Number of reduce tasks is set to 0 since there's no reduce operator  
org.apache.hadoop.security.AccessControlException: Permission denied:  
user=jbae, access=WRITE,  
inode="/hive/warehouse":hadoop:supergroup:drwxr-xr-x  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5507)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5489)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5463)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3606)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3576)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3550)  
at  
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)  
at  
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)  
at  
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)  
at  
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)  
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at  
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)  
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)  
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native  
Method)  
at  
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)  
at  
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)  
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)  
at  
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)  
at  
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)  
at  
org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2555)  
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2524)  
at  
org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:827)  
at  
org.apache.hadoop.hdfs.DistributedFileSystem$16.doCall(DistributedFileSystem.java:823)  
at  
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)  
at  
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:823)  
at  
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:816)  
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1863)  
at  
org.apache.hadoop.hive.ql.exec.ExecDriver.createTmpDirs(ExecDriver.java:250)  
at  
org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:458)  
at  
org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:138)  
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:144)  
at  
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)  
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)  
Caused by:  
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):  
Permission denied: user=jbae, access=WRITE,  
inode="/hive/warehouse":hadoop:supergroup:drwxr-xr-x  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5507)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5489)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5463)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3606)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3576)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3550)  
at  
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)  
at  
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)  
at  
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)  
at  
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)  
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at  
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)  
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)

```
    at org.apache.hadoop.ipc.Client.call(Client.java:1410)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at 

```

org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:215)  
at com.sun.proxy.$Proxy32.mkdirs(Unknown Source)  
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  
at  
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)  
at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)  
at java.lang.reflect.Method.invoke(Method.java:606)  
at  
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:190)  
at  
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)  
at com.sun.proxy.$Proxy32.mkdirs(Unknown Source)  
at  
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:500)  
at  
org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2553)  
... 13 more  
Job Submission failed with exception  
'org.apache.hadoop.security.AccessControlException(Permission denied:  
user=jbae, access=WRITE,  
inode="/hive/warehouse":hadoop:supergroup:drwxr-xr-x  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkFsPermission(FSPermissionChecker.java:265)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:251)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:232)  
at  
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:176)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5507)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkPermission(FSNamesystem.java:5489)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkAncestorAccess(FSNamesystem.java:5463)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInternal(FSNamesystem.java:3606)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirsInt(FSNamesystem.java:3576)  
at  
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3550)  
at  
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:754)  
at  
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:558)  
at  
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)  
at  
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:599)  
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)  
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)  
at java.security.AccessController.doPrivileged(Native Method)  
at javax.security.auth.Subject.doAs(Subject.java:415)  
at  
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)  
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)  
)'  
FAILED: Execution Error, return code 1 from  
org.apache.hadoop.hive.ql.exec.MapRedTask

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e05325bb-40d7-496b-bc4b-ff1984439f70%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e05325bb-40d7-496b-bc4b-ff1984439f70%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:04am UTC](https://discuss.elastic.co/t/hadoop-running-elasticsearch-hive-in-aws-emr/19579/2 "2017-07-06T01:04:51Z")

</div>