My mistake. It is es.nodes not es.resource.nodes
There is no such property 'es.resource.node' used in Es-Hadoop.
And es.resource.write is something set internally by the Hive support not
to mention it should point to a index/type NOT
host:port.
You most likely want:
es.nodes=eshost:9200
and
es.resource=[index]/[type]
See the previous link that I've listed on the properties supported:
[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic
On 3/15/2014 1:48 AM, P lva wrote:
I got it to work by replacing
'es.resource.write'='http://eshost:9200'
'es.resource.node'='http://eshost:9200'
as one of the tbl properties in hive.
Thanks for your help Costin.
On Friday, March 14, 2014 1:12:00 PM UTC-5, Costin Leau wrote:
You should not change the configuration in Elasticsearch. By
default, ES binds on all available interfaces - specifying
an Ip:port is likely to restrict access rather than extend it.
To find out the configuration options in es-hadoop, look no further
than the docs [1]. If the hive job is running on
the
same machine as elasticsearch, the default (localhost:9200) should
be enough. As an alternative you can specify the
public ip of your host: x.y.z.w
As a last step, enable TRACE logging (through log4j) on
org.elasticsearch.hadoop package and see what comes out
including network connectivity.
By the way, make sure you don't have any firewall or proxy set on
your system which might be picked up automatically by
the JVM.
[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>
On 3/14/2014 6:50 PM, P lva wrote:
> Costin,
>
> About what you asked,
> 1) Hive server is running on the same machine as elasticsearch,
> 2) I get a response when I do a
curlhttp://elasticsearc&hivehost:9200
>
>
> I feel I'm missing something simple. This is what I've got until
now.
>
> 1) First error (connection refused) is because i left the default
settings as is.
>
> I changed the 'network.host' in elasticsearch.yml to the hostname
>
> 2) Second error (cannot discover elasticsearch version) is when I
changed the 'network.host' in elasticsearch.yml to the
> hostname.
>
> Looked at the source code and figured I'm expected to pass
eshost:port to the hive table at table creation.
> (
https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48
<
https://github.com/elasticsearch/elasticsearch-hadoop/blob/master/src/main/java/org/elasticsearch/hadoop/hive/EsStorageHandler.java#L48>)
> .
> So I included 'es.resource.write'='http://elasticsearchhost:9200 <
http://elasticsearchhost:9200>' as one of the table properties during
table creation
step.
>
> STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
> TBLPROPERTIES('es.resource' = 'esdemo/hive',
> 'es.mapping.name <http://es.mapping.name>'='time:@timestamp',
> 'es.resource.write'='http://eshost:9200');
>
> Now I get connection refused again. Is that the right way to pass
that information ?
>
> Thanks for you patience and help
>
>
>
>
> On Friday, March 14, 2014 10:33:14 AM UTC-5, Costin Leau wrote:
>
> Without any extra information I'm afraid I can only guess what
might be the issue.
> Make sure you have the latest Elasticsearch 0.90 or 1.x
available on port 9200 with the HTTP/REST port open.
> Also make sure that Hive actually runs on the same machine -
not just the client but also the server (meaning Hadoop
> itself).
>
> You indicate that if you change the network configuration you
get an error regarding the version - this suggests that:
>
> 1. Hive is actually running on a different machine than ES -
hence the network error
> 2. After pointing Hive to the actual ES machine, you get an
error since you're using an old Elasticsearch version
> (0.20)
>
> Cheers,
>
> On 3/14/2014 12:19 AM, P lva wrote:
> > I have a simple query
> > insert into table eslogs select * from eslogs_ext;
> > Hive and elasticsearch are running on the same host.
> >
> > To execute the script I'm following the directions from the
link.
> >
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>
> <
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>>
> >
> > There are two elasticsearch nodes, and they can recognize
each other (as indicated by start up process) , but why would
> > hive not be able to pick them up ? Can you explain what
could have gone wrong ?
> >
> >
> > On Thursday, March 13, 2014 4:28:14 PM UTC-5, Costin Leau
wrote:
> >
> > What does your Hive script look like? Can you confirm
the ip/address of your Hive and Elasticsearch ? How are you
> > executing the script?
> > The error indicates an error in your network
configuration.
> >
> > Cheers,
> >
> > P.S. Feel free to post a gist or whatever it's
convenient.
> >
> > On 3/13/2014 10:38 PM, P lva wrote:
> > > Hi, I have few weblogs in a hive table that I'd like
to visualize in kibana.
> > > ES is on the same node as hive server.
> > >
> > > Followed directions from this pagehttp://
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>
> <
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>>
> > <
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>
> <
Elasticsearch Platform — Find real-time answers at scale | Elastic
<
Elasticsearch Platform — Find real-time answers at scale | Elastic>>>
> > >
> > > I can create a table using esstorage handler, but
when I tried to ingest data into this table I got
> > >
> > > Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing
> > > row {***first row of my table**}
> > > at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:175)
> > > at
org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> > > at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
> > > at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> > > at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
> > > at
java.security.AccessController.doPrivileged(Native Method)
> > > at
javax.security.auth.Subject.doAs(Subject.java:415)
> > > at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> > > at
org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
> > > Caused by:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {*** first row of
> > > my table**}
> > > Caused by:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: Out
of nodes and retries; caught exception
> > > at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:652)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > > at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:88)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > > at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:91)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
> > > at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:842)
> > > at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:534)
> > > ... 9 more
> > > Caused by: java.io.IOException: Out of nodes and
retries; caught exception
> > > at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:81)
> > > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:221)
> > > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:205)
> > > at
org.elasticsearch.hadoop.rest.RestClient.execute(RestClient.java:209)
> > > at
org.elasticsearch.hadoop.rest.RestClient.get(RestClient.java:103)
> > > at
org.elasticsearch.hadoop.rest.RestClient.discoverNodes(RestClient.java:85)
> > > at
org.elasticsearch.hadoop.rest.InitializationUtils.discoverNodesIfNeeded(InitializationUtils.java:60)
> > > at
org.elasticsearch.hadoop.mr.EsOutputFormat$ESRecordWriter.init(EsOutputFormat.java:165)
> > > at
org.elasticsearch.hadoop.hive.EsHiveOutputFormat$ESHiveRecordWriter.write(EsHiveOutputFormat.java:50)
> > > at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:638)
> > > ... 18 more
> > > Caused by: java.net.ConnectException: Connection
refused
> > > at
java.net.PlainSocketImpl.socketConnect(Native Method)
> > > at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> > > at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> > > at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> > > at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > at java.net.Socket.connect(Socket.java:579)
> > > at java.net.Socket.connect(Socket.java:528)
> > > at java.net.Socket.<init>(Socket.java:425)
> > > at java.net.Socket.<init>(Socket.java:280)
> > > at
> > >
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:80)
> > > at
> > >
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:122)
> > > at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
> > > at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
> > > at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
> > > at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
> > > at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
> > > at
org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport.execute(CommonsHttpTransport.java:160)
> > > at
org.elasticsearch.hadoop.rest.NetworkClient.execute(NetworkClient.java:74)
> > > ... 27 more
> > >
> > > Now, I changed the config network.host to the ipadress
of the server. Now when I run hive insert data i get
> > > FAILED: IllegalStateException Cannot discover
Elasticsearch version
> > >
> > >
> > > --
> > > You received this message because you are subscribed
to the Google Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving
emails from it, send an email to
> > >elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:> <javascript:>
<javascript:>
> <javascript:>>.
> > > To view this discussion on the web visit
> > >
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>
> <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>>
> > <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>
> <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com>>>
> > > <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>
> <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>>
>
> > <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>
> <
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/a9878590-71b9-46e5-8bd7-7ee3878f959e%40googlegroups.com?utm_medium=email&utm_source=footer>>>>.
>
> >
> > > For more options, visithttps://
groups.google.com/d/optout http://groups.google.com/d/optout <
http://groups.google.com/d/optout
<http://groups.google.com/d/optout>> <
https://groups.google.com/d/optout https://groups.google.com/d/optout
> <https://groups.google.com/d/optout <
https://groups.google.com/d/optout>>>.
> >
> > --
> > Costin
> >
> > --
> > You received this message because you are subscribed to the
Google Groups "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails
from it, send an email to
> >elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:> <javascript:>
<javascript:>>.
> > To view this discussion on the web visit
> >
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com>
> <
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com>>
> > <
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer>
> <
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/97ef1333-eb9d-43d7-a3fe-b5e48d072643%40googlegroups.com?utm_medium=email&utm_source=footer>>>.
>
> > For more options, visithttps://groups.google.com/d/optout <
http://groups.google.com/d/optout> <https://groups.google.com/d/optout
<https://groups.google.com/d/optout>>.
>
> --
> Costin
>
> --
> You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it,
send an email to
>elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:> <javascript:>>.
> To view this discussion on the web visit
>
https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com>
> <
https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com?utm_medium=email&utm_source=footer
<
https://groups.google.com/d/msgid/elasticsearch/d1aa80dc-d45b-4372-b287-f7a0c3e8bb46%40googlegroups.com?utm_medium=email&utm_source=footer>>.
> For more options, visithttps://groups.google.com/d/optout <
https://groups.google.com/d/optout>.
--
Costin
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9041befe-dad6-447f-b666-c3384aa32259%40googlegroups.com
<
https://groups.google.com/d/msgid/elasticsearch/9041befe-dad6-447f-b666-c3384aa32259%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout.
--
Costin