Problem with ES- Shield

ElasticSearch_Users_ · April 3, 2015, 1:05pm

I've tried this :
create external table es_word(word string,cnt bigint)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'secure/secure',
'es.nodes'='192.168.238.130:9200',
'es.net.http.auth.user'='myysername',
'es.net.http.auth.pass'='mypassword');

INSERT OVERWRITE TABLE es_word SELECT s.word, s.cnt FROM word_countstest s;

But the job is terminated with an authentication error :
Any idea why? I've entered the thr right esuser and pwd that I've defined
and am using to access to my ES cluster !

Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {"word":"sur","cnt":3}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row {"word":"sur","cnt":3}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:544)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:157)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:
AuthenticationException[missing authentication token for REST request
[/]]
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:286)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:516)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:572)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
... 9 more
Caused by: org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest:
AuthenticationException[missing authentication token for REST request
[/]]
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:319)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:284)
at org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:288)
at org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:117)
at org.elasticsearch.hadoop.rest.RestClient.esVersion(RestClient.java:364)
at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:86)
at org.elasticsearch.hadoop.hive.HiveUtils.init(HiveUtils.java:145)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:93)
at org.elasticsearch.hadoop.hive.EsHiveOutputFormat.getHiveRecordWriter(EsHiveOutputFormat.java:42)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:296)
at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:283)
... 20 more

ᐧ

On Thu, Apr 2, 2015 at 9:59 PM, BEN SALEM Omar omar.bensalem@esprit.tn
wrote:

Hey Cosin,
And when It comes to the marvel?
Can you see why It's not seeing my indices?
Thanks,

On Thu, Apr 2, 2015 at 9:46 PM, Costin Leau costin.leau@gmail.com wrote:
The configuration example is correct - one can pass any configuration
property there and it will be picked up as explained here [1]

Additionally, to verify one can turn on logging [2] to see the connection
is properly authenticated

[1] Elasticsearch for Apache Hadoop and Spark | Elastic
master/pig.html#pig-configuration
[2] Elasticsearch for Apache Hadoop and Spark | Elastic
master/logging.html

On 4/2/15 9:21 PM, Jay Modi wrote:
Still not sure what's going on with Marvel, unfortunately. No idea if
this will help, but have you tried clearing your
browser cache or using a incognito window?

I think something like this:

TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200 http://192.168.238.130:9200/',
'es.net.http.auth.user'='omar',
'es.net.http.auth.pass'='omar')

I copy/pasted the wrong names earlier. I'm not a hadoop user but I
believe that should work based off of the following
documentation:
Using Elasticsearch for Apache Hadoop with Shield | Shield [2.4] | Elastic
Elasticsearch for Apache Hadoop and Spark | Elastic
current/hive.html#hive-configuration

On Thursday, April 2, 2015 at 2:05:15 PM UTC-4, BEN SALEM Omar wrote:
Jay,
when it comes to adding the .user and .password in the tblproperties
Do you know what exactly to do?
let's suppose that my admin esuser is omar and pwd omar
what should I do?
TBLPROPERTIES('es.resource' = 't/t',
'es.nodes'='192.168.238.130:9200 <http://192.168.238.130:9200>',
                 and now what to write);
and is there any further config to do?
Thanks for your time
ᐧ

On Thu, Apr 2, 2015 at 7:29 PM, BEN SALEM Omar <omar.b...@esprit.tn
<javascript:>> wrote:
    Yes I had,
    I used to manage my cluster with marvel perfectly and all the
indices used to show up.
But now, none of them seem to be found due to time span; and
even when I removed the shield this "porblem" occured.
I don't see how to "restore" the previous configuration.
Any idea?
ᐧ
    On Thu, Apr 2, 2015 at 7:14 PM, Jay Modi <j...@elastic.co
<javascript:>> wrote:
        There should not be anything extra necessary to work with
Marvel. The browser prompted you for credentials
and you entered admin credentials, correct?
        On Thursday, April 2, 2015 at 12:39:03 PM UTC-4, BEN SALEM
Omar wrote:
            Or not, that wenti as I'd like to go :))
            Here is the last remaining issue :


            
            ᐧ

            On Thu, Apr 2, 2015 at 6:32 PM, BEN SALEM Omar <
omar.b...@esprit.tn> wrote:
                That would be a problem !
                Last question (as for now :p ), how to change the
marvel timepicker? I can't see any indice of
mine(as I already told you)
And hey, thank you for your cooperation Jay
ᐧ
                On Thu, Apr 2, 2015 at 6:26 PM, Jay Modi <
j...@elastic.co> wrote:
                    Hi Omar,

                    I think you need to add "es.net.http.auth.user"
and "es.net.http.auth_password" as
additional TBLPROPERTIES. You will need to using
elasticsearch Hadoop 2.1 beta 3 for this to
work as far as I know.
                    -Jay

                    On Thursday, April 2, 2015 at 11:18:12 AM UTC-4,
BEN SALEM Omar wrote:
                        Hi Jay,
                        Now that I have shield in both ES and
Kibana, and I want to load some data from my hadoop
cluster to ES.
I usually do this :
....
....
create external table es_word_count(word
string, count int)
STORED BY 'org.elasticsearch.hadoop.__
hive__.EsStorageHandler'
TBLPROPERTIES('es.resource' = 'test/test',
'es.nodes'='192.168.238.130 <
http://192.168.238.130>:__92__00');
INSERT OVERWRITE TABLE es_word_count SELECT
s.word, s.count FROM word_counts1 s;
                        what should I do or add to access to my ES
cluster as the esuser I've created?
Thanks
ᐧ
                        On Thu, Apr 2, 2015 at 1:08 PM, Jay Modi <
j...@elastic.co> wrote:
                            Hi Omar,

                            Have you configured Kibana and Marvel to
work with Shield? For Kibana 4, please see
http://www.elastic.co/__guid__
e/en/kibana/current/production.html#configuring-__kibana-__shield
<Starting with the Elasticsearch Platform and its Solutions | Elastic
en/kibana/current/production.html#configuring-kibana-shield> and
http://www.elastic.co/__guid__
e/en/shield/current/___shield___with_kibana_4.html
<Starting with the Elasticsearch Platform and its Solutions | Elastic
en/shield/current/_shield_with_kibana_4.html>. For Marvel,
see http://www.elastic.co/__guid__
e/en/shield/current/__marvel.__html
<Starting with the Elasticsearch Platform and its Solutions | Elastic
en/shield/current/marvel.html>
                            -Jay

                            On Thursday, April 2, 2015 at 6:45:55 AM
UTC-4, BEN SALEM Omar wrote:
                                Hi Jay,

                                I have a question, now that I've
added an esuser as an admin, I can not run my
Kibana anymore,
                                Any idea on what to do to overcome
this?
                                Another basic question, when I open
the marvel, it shows no indice because the
sample of data I'm working with is
in "2014", how to change my time span to see my
indices again?
btwn, before I install the shield,
Marvel used to recognize my timestamp by default
                                
                                ᐧ
                                Thank you,
                                Omar,

                                On Wed, Apr 1, 2015 at 8:22 PM, BEN
SALEM Omar omar.b...@esprit.tn wrote:
                                    When I try to add an esuser and
grant him a role:
                                    bin/shield/esusers useradd
rdeniro-p taxidriver-r admin
                                    It says that there is an error
in the "esusers" file in the elef['$ES_INCLUDE"]
line, I think the line 31 or 38,
I forgot.
In the line above, there is a
$...done: which ends with ":", if you change it to
done; with ";" , the error
doesn't show up anymore.
                                    I'm installing the whole thing
in the Hortonworks Sandbox 2.0 vm
                                    On Wed, Apr 1, 2015 at 8:14 PM,
Jay Modi j...@elastic.co wrote:
                                        I'm glad you were able to
fix this issue. I have a few questions that may
help us ensure this fix is
pulled back into Shield. What OS and Shell are
you using? Which line in the
script did you change exactly? I'm looking at
the script right now and I
don't see any instance of "done:"
                                        Just an FYI, there is
another way to handle the configuration directory
issue that is documented in
the configuring your environment section of the
shield guide:

http://www.elastic.co/guide/__en____/shield/current/getting-
__starte____d.html#_configuring___your___envir__onment
<
http://www.elastic.co/guide/en/shield/current/getting-
started.html#_configuring_your_environment>
                                        On Wednesday, April 1, 2015
at 2:07:28 PM UTC-4, BEN SALEM Omar wrote:
                                            Ok, I've managed to
solve my own problem.
                                            First of all things, if
you want to just remove the shield plugin run
under the elasticsearch :
                                            $ bin/plugin --remove
shield
                                            Now, to the syntax error
in the elef line in the "esusers", generally,
when there is such an
error, it's related to things before that line.
In fact there was
something like this :
$....done:
correction :
$....done;
                                            Now, we could add users
within shield and grunt them the desired role.
                                            But, even if I've done
it, I couldn't access to my ES cluster with the
"new born user"; I think
because my ES is running as a service and not
starting with
bin/elasticsearch, that lead to "misunderstanding."
                                            The solution I found was
to transfer the

/usr/share/elasticsearch/__confi______g/shield under /etc/elasticsearch
$ln -s
/usr/share/elasticsearch/__confi______g/shield/ /etc/elasticsearch
                                            And that was it !
                                            Hope that could be of a
help for anyone else facing the same problem !
                                            On Wednesday, April 1,
2015 at 5:28:14 PM UTC+2, BEN SALEM Omar wrote:
                                                I wanted to add
authentication to my ES, thus I downloaded the shield :

https://www.elastic.co/__downloa______ds/shield
<
Download Shield | Security for Elasticsearch>
                                                Then I've tried to
add a user to give him the admin role as it's
mentionned here :

http://www.elastic.co/guide/__en______/shield/current/
getting-__starte______d.html
<
Getting Started with Shield | Shield [2.4] | Elastic>
(I've added it with
root, cause I start my es with root..)
                                                But when I try to do
so, an error occure saying that there is a
syntax error in the
38 line "the elef line"
elef[$ES_INCLUDE]
bla bla
I've tried many time
:
bin/shield/esusers
useradd omar -p omar -r admin
bin/shield/esusers
useradd esomar -p omar -r admin
bin/shield/esusers
useradd es_omar -p omar -r admin
bin/shield/esusers
useradd ES_omar -p omar -r admin
bin/shield/esusers
useradd omar -r admin
bin/shield/esusers
useradd esomar -r admin
bin/shield/esusers
useradd es_omar -r admin
bin/shield/esusers
useradd ES_omar -r admin
                                                But the same error
kept on jumping !
                                                I said enough is
enough and wanted to keep on doing wwhat I was
doing, tried to run
Kibana, and this is what happened !
                                                I've then tried to
open my ES marvel plugin, but there is now an
authentication box
                                                
                                                I've tried ALL the
combinations above, but none of them seemed to
work !!
                                                I'm really
preoccupied by this note :
To ensure that
Elasticsearch can read the user and role information
at startup, run
|esusers useradd| as the same user you use to run
Elasticsearch.
Running the command as root or some other user will
update the
permissions for the |users| and

|users_roles|__file______s and prevent Elasticsearch from accessing
them.
                                                The thing is that I
start my es and kibana with the root..
Please HELP !!
                                        --
                                        You received this message
because you are subscribed to a topic in the
Google Groups
"elasticsearch" group.
To unsubscribe from this
topic, visit

https://groups.google.com/d/__to____pic/elasticsearch/
sEZpCO7E___B0/____unsubscribe
<
https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_B0/unsubscribe

.
To unsubscribe from this
group and all its topics, send an email to
elasticsearc...@__
googlegroups.__c__om.
To view this discussion on
the web visit

https://groups.google.com/d/__ms____gid/elasticsearch/
9f0ebe63-__cc8____3-4c0b-8ee9-1514e51f77f6%__40goo____glegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/9f0ebe63-
cc83-4c0b-8ee9-1514e51f77f6%40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players
email&utm_source=footer>.
For more options, visit
https://groups.google.com/d/__op____tout
<
https://groups.google.com/d/optout>.
                            --
                            You received this message because you
are subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__
to__pic/elasticsearch/sEZpCO7E___B0/__unsubscribe
<https://groups.google.com/d/
topic/elasticsearch/sEZpCO7E_B0/unsubscribe>.
To unsubscribe from this group and all
its topics, send an email to
elasticsearc...@__googlegroups.__com.
To view this discussion on the web visit
https://groups.google.com/d/__
ms__gid/elasticsearch/35767d7f-__45d__7-4113-a478-499d44abcf38%40goo
glegroups.com
<https://groups.google.com/d/
msgid/elasticsearch/35767d7f-45d7-4113-a478-499d44abcf38%
40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players>.
                            For more options, visit
https://groups.google.com/d/__op__tout
https://groups.google.com/d/optout.
                    --
                    You received this message because you are
subscribed to a topic in the Google Groups
"elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/__
topic/elasticsearch/sEZpCO7E___B0/unsubscribe
<https://groups.google.com/d/
topic/elasticsearch/sEZpCO7E_B0/unsubscribe>.
To unsubscribe from this group and all its
topics, send an email to
elasticsearc...@__googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/__
msgid/elasticsearch/579d2139-_a75f-4c17-a03e-51e0f4830faf%
_40googlegroups.com
<https://groups.google.com/d/
msgid/elasticsearch/579d2139-a75f-4c17-a03e-51e0f4830faf%
40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players>.
                    For more options, visit
https://groups.google.com/d/__optout <https://groups.google.com/d/optout

.
        --
        You received this message because you are subscribed to a
topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_
B0/unsubscribe
<https://groups.google.com/d/topic/elasticsearch/sEZpCO7E_
B0/unsubscribe>.
To unsubscribe from this group and all its topics, send an
email to elasticsearc...@googlegroups.com
<javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9622bb08-
27c8-45b0-a892-b7412ca8fdff%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/9622bb08-
27c8-45b0-a892-b7412ca8fdff%40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players
email&utm_source=footer>.
        For more options, visit https://groups.google.com/d/optout <
https://groups.google.com/d/optout>.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch+
unsubscribe@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e830a27c-
a73a-447f-bbbf-58c7a9e31957%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/e830a27c-
a73a-447f-bbbf-58c7a9e31957%40GGGROUPS CASINO – Real Slot Casino for 10,000+ Senior Players
email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/sEZpCO7E_B0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/551D9CA9.6040809%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGgBrpxW801AgUXqs_WAZEqgYe3KjZiZ2sOy6dja-J4nXQ5Cuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
URGENT HELP : can't connect to my ES and Kibana after installing shield! Elasticsearch	2	618	July 6, 2017
Shield is blocking marvel Elasticsearch elastic-stack-security	2	1024	January 12, 2017
[Hadoop] Specifying username/password information for Shield-configured Elasticsearch Elasticsearch	7	515	July 6, 2017
Elasticsearch shield relam problem Elasticsearch	9	683	July 6, 2017
How to make unstructured data available in Hadoop for analysis with Elasticsearch? Elasticsearch	10	1883	July 6, 2017

Problem with ES- Shield

Related topics