Turn on logging in es-hadoop on the REST package [1]. This will indicate the type of connection made and any
authentication scheme applied and its result.
[1] Logging | Elasticsearch for Apache Hadoop [master] | Elastic
On 4/3/15 4:05 PM, 'BEN SALEM Omar' via elasticsearch wrote:
I've tried this :
create external table es_word(word string,cnt bigint)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'secure/secure',
'es.nodes'='192.168.238.130:9200 http://192.168.238.130:9200',
'es.net.http.auth.user'='myysername',
'es.net.http.auth.pass'='mypassword');
INSERT OVERWRITE TABLE es_word SELECT s.word, s.cnt FROM word_countstest s;But the job is terminated with an authentication error :
Any idea why? I've entered the thr right esuser and pwd that I've defined and am using to access to my ES cluster !
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"word":"sur","cnt":3}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"word":"sur","cnt":3}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:544)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: AuthenticationException[missing authentication token for REST request [/]]
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:286)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:516)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:572)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
... 9 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: AuthenticationException[missing authentication token for REST request [/]]
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:319)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:284)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:288)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:117)
at org.elasticsearch.hadoop.rest.RestClient.esVersion(RestClient.java:364)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:86)
at org.elasticsearch.hadoop.hive.HiveUtils.init(HiveUtils.java:145)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:93)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:42)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:296)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:283)
... 20 moreᐧ
On Thu, Apr 2, 2015 at 9:59 PM, BEN SALEM Omar <omar.bensalem@esprit.tn mailto:omar.bensalem@esprit.tn> wrote:
Hey Cosin, And when It comes to the marvel? Can you see why It's not seeing my indices? Thanks, On Thu, Apr 2, 2015 at 9:46 PM, Costin Leau <costin.leau@gmail.com <mailto:costin.leau@gmail.com>> wrote: The configuration example is correct - one can pass any configuration property there and it will be picked up as explained here [1] Additionally, to verify one can turn on logging [2] to see the connection is properly authenticated [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/pig.html#pig-configuration [2] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html On 4/2/15 9:21 PM, Jay Modi wrote: Still not sure what's going on with Marvel, unfortunately. No idea if this will help, but have you tried clearing your browser cache or using a incognito window? I think something like this: TBLPROPERTIES('es.resource' = 't/t', 'es.nodes'='192.168.238.130:9200 <http://192.168.238.130:9200> <http://192.168.238.130:9200/>', 'es.net.http.auth.user'='omar', 'es.net.http.auth.pass'='omar') I copy/pasted the wrong names earlier. I'm not a hadoop user but I believe that should work based off of the following documentation: http://www.elastic.co/guide/en/shield/current/hadoop.html http://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html#hive-configuration On Thursday, April 2, 2015 at 2:05:15 PM UTC-4, BEN SALEM Omar wrote: Jay, when it comes to adding the .user and .password in the tblproperties Do you know what exactly to do? let's suppose that my admin esuser is omar and pwd omar what should I do? TBLPROPERTIES('es.resource' = 't/t', 'es.nodes'='192.168.238.130:9200 <http://192.168.238.130:9200> <http://192.168.238.130:9200>', and now what to write); and is there any further config to do? Thanks for your time ᐧ On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar <omar.b...@esprit.tn <mailto:omar.b...@esprit.tn> <javascript:>> wrote: Yes I had, I used to manage my cluster with marvel perfectly and all the indices used to show up. But now, none of them seem to be found due to time span; and even when I removed the shield this "porblem" occured. I don't see how to "restore" the previous configuration. Any idea? ᐧ On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi <j...@elastic.co <mailto:j...@elastic.co> <javascript:>> wrote: There should not be anything extra necessary to work with Marvel. The browser prompted you for credentials and you entered admin credentials, correct? On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM Omar wrote: Or not, that wenti as I'd like to go :)) Here is the last remaining issue : ᐧ On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar <omar.b...@esprit.tn <mailto:omar.b...@esprit.tn>> wrote: That would be a problem ! Last question (as for now :p ), how to change the marvel timepicker? I can't see any indice of mine(as I already told you) And hey, thank you for your cooperation Jay ᐧ On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi <j...@elastic.co <mailto:j...@elastic.co>> wrote: Hi Omar, I think you need to add "es.net.http.auth.user" and "es.net.http.auth_password" as additional TBLPROPERTIES. You will need to using elasticsearch Hadoop 2.1 beta 3 for this to work as far as I know. -Jay On Thursday, April 2, 2015 at 11:18:12 AM UTC-4, BEN SALEM Omar wrote: Hi Jay, Now that I have shield in both ES and Kibana, and I want to load some data from my hadoop cluster to ES. I usually do this : .... .... create external table es_word_count(word string, count int) STORED BY 'org.elasticsearch.hadoop.__hive__.EsStorageHandler' TBLPROPERTIES('es.resource' = 'test/test', 'es.nodes'='192.168.238.130 <http://192.168.238.130>:__92__00'); INSERT OVERWRITE TABLE es_word_count SELECT s.word, s.count FROM word_counts1 s; what should I do or add to access to my ES cluster as the esuser I've created? Thanks ᐧ On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi <j...@elastic.co <mailto:j...@elastic.co>> wrote: Hi Omar, Have you configured Kibana and Marvel to work with Shield? For Kibana 4, please see http://www.elastic.co/__guid__e/en/kibana/current/__production__.html#configuring-__kibana-__shield <http://www.elastic.co/guide/en/kibana/current/production.html#configuring-kibana-shield> and http://www.elastic.co/__guid__e/en/shield/current/___shield___with_kibana_4.html <http://www.elastic.co/guide/en/shield/current/_shield_with_kibana_4.html>. For Marvel, see http://www.elastic.co/__guid__e/en/shield/current/__marvel.__html <http://www.elastic.co/guide/en/shield/current/marvel.html> -Jay On Thursday, April 2, 2015 at 6:45:55 AM UTC-4, BEN SALEM Omar wrote: Hi Jay, I have a question, now that I've added an esuser as an admin, I can not run my Kibana anymore, Any idea on what to do to overcome this? Another basic question, when I open the marvel, it shows no indice because the sample of data I'm working with is in "2014", how to change my time span to see my indices again? btwn, before I install the shield, Marvel used to recognize my timestamp by default ᐧ Thank you, Omar, On Wed, Apr 1, 2015 at 8:22 PM, BEN SALEM Omar <omar.b...@esprit.tn <mailto:omar.b...@esprit.tn>> wrote: When I try to add an esuser and grant him a role: bin/shield/esusers useradd rdeniro-p taxidriver-r admin It says that there is an error in the "esusers" file in the elef['$ES_INCLUDE"] line, I think the line 31 or 38, I forgot. In the line above, there is a $...done: which ends with ":", if you change it to done; with ";" , the error doesn't show up anymore. I'm installing the whole thing in the Hortonworks Sandbox 2.0 vm On Wed, Apr 1, 2015 at 8:14 PM, Jay Modi <j...@elastic.co <mailto:j...@elastic.co>> wrote: I'm glad you were able to fix this issue. I have a few questions that may help us ensure this fix is pulled back into Shield. What OS and Shell are you using? Which line in the script did you change exactly? I'm looking at the script right now and I don't see any instance of "done:" Just an FYI, there is another way to handle the configuration directory issue that is documented in the configuring your environment section of the shield guide: http://www.elastic.co/guide/__en____/shield/current/getting-__starte____d.html#_configuring___your___envir__onment <http://www.elastic.co/guide/en/shield/current/getting-started.html#_configuring_your_environment> On Wednesday, April 1, 2015 at 2:07:28 PM UTC-4, BEN SALEM Omar wrote: Ok, I've managed to solve my own problem. First of all things, if you want to just remove the shield plugin run under the elasticsearch : $ bin/plugin --remove shield Now, to the syntax error in the elef line in the "esusers", generally, when there is such an error, it's related to things before that line. In fact there was something like this : $....done: correction : $....done; Now, we could add users within shield and grunt them the desired role. But, even if I've done it, I couldn't access to my ES cluster with the "new born user"; I think because my ES is running as a service and not starting with bin/elasticsearch, that lead to "misunderstanding." The solution I found was to transfer the /usr/share/elasticsearch/__confi______g/shield under /etc/elasticsearch $ln -s /usr/share/elasticsearch/__confi______g/shield/ /etc/elasticsearch And that was it ! Hope that could be of a help for anyone else facing the same problem ! On Wednesday, April 1, 2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote: I wanted to add authentication to my ES, thus I downloaded the shield : https://www.elastic.co/__downloa______ds/shield <https://www.elastic.co/downloads/shield> Then I've tried to add a user to give him the admin role as it's mentionned here : http://www.elastic.co/guide/__en______/shield/current/getting-__starte______d.html <http://www.elastic.co/guide/en/shield/current/getting-started.html> (I've added it with root, cause I start my es with root..) But when I try to do so, an error occure saying that there is a syntax error in the 38 line "the elef line" elef[$ES_INCLUDE] bla bla I've tried many time : bin/shield/esusers useradd omar -p omar -r admin bin/shield/esusers useradd esomar -p omar -r admin bin/shield/esusers useradd es_omar -p omar -r admin bin/shield/esusers useradd ES_omar -p omar -r admin bin/shield/esusers useradd omar -r admin bin/shield/esusers useradd esomar -r admin bin/shield/esusers useradd es_omar -r admin bin/shield/esusers useradd ES_omar -r admin But the same error kept on jumping ! I said enough is enough and wanted to keep on doing wwhat I was doing, tried to run Kibana, and this is what happened ! I've then tried to open my ES marvel plugin, but there is now an authentication box I've tried ALL the combinations above, but none of them seemed to work !! I'm really preoccupied by this note : To ensure that Elasticsearch can read the user and role information at startup, run |esusers useradd| as the same user you use to run Elasticsearch. Running the command as root or some other user will update the permissions for the |users| and |users_roles|__file______s and prevent Elasticsearch from accessing them. The thing is that I start my es and kibana with the root.. Please HELP !! -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/__to____pic/elasticsearch/sEZpCO7E___B0/____unsubscribe <https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>. To unsubscribe from this group and all its topics, send an email to elasticsearc...@__googlegroups.__c__om. To view this discussion on the web visit https://groups.google.com/d/__ms____gid/elasticsearch/9f0ebe63-__cc8____3-4c0b-8ee9-1514e51f77f6%__40goo____glegroups.com <https://groups.google.com/d/msgid/elasticsearch/9f0ebe63-cc83-4c0b-8ee9-1514e51f77f6%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/__op____tout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/__to__pic/elasticsearch/sEZpCO7E___B0/__unsubscribe <https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>. To unsubscribe from this group and all its topics, send an email to elasticsearc...@__googlegroups.__com. To view this discussion on the web visit https://groups.google.com/d/__ms__gid/elasticsearch/35767d7f-__45d__7-4113-a478-499d44abcf38%__40goo__glegroups.com <https://groups.google.com/d/msgid/elasticsearch/35767d7f-45d7-4113-a478-499d44abcf38%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/__op__tout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/__topic/elasticsearch/sEZpCO7E___B0/unsubscribe <https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>. To unsubscribe from this group and all its topics, send an email to elasticsearc...@__googlegroups.com <http://googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/__msgid/elasticsearch/579d2139-__a75f-4c17-a03e-51e0f4830faf%__40googlegroups.com <https://groups.google.com/d/msgid/elasticsearch/579d2139-a75f-4c17-a03e-51e0f4830faf%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/__optout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe <https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe>. To unsubscribe from this group and all its topics, send an email to elasticsearc...@googlegroups.com <mailto:elasticsearc...@googlegroups.com> <javascript:>. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9622bb08-27c8-45b0-a892-b7412ca8fdff%40googlegroups.com <https://groups.google.com/d/msgid/elasticsearch/9622bb08-27c8-45b0-a892-b7412ca8fdff%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com> <mailto:elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>>. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e830a27c-a73a-447f-bbbf-58c7a9e31957%40googlegroups.com <https://groups.google.com/d/msgid/elasticsearch/e830a27c-a73a-447f-bbbf-58c7a9e31957%40googlegroups.com?utm_medium=email&utm_source=footer>. For more options, visit https://groups.google.com/d/optout. -- Costin -- You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe. To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551D9CA9.6040809%40gmail.com. For more options, visit https://groups.google.com/d/optout.--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGgBrpxW801AgUXqs_WAZEqgYe3KjZiZ2sOy6dja-J4nXQ5Cuw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGgBrpxW801AgUXqs_WAZEqgYe3KjZiZ2sOy6dja-J4nXQ5Cuw%40mail.gmail.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.
--
Costin
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/551E96D6.10001%40gmail.com.
For more options, visit https://groups.google.com/d/optout.