[hadoop] Reading severals index with Hive

Hi,

I have got severals index in my ElasticSearch starting with log-access. I
would like to read all of them thanks to Hive.

I have tried 'es.resource' = 'log-access*/access' which doesn't seem to be
correct.

Then i have tried " 'es.resource' =
'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' =
'log-access/{@timestamp:YYYY.MM.dd}'.
It resulted in the error below even though i have managed reading one index

Connection error (check network and/or proxy settings)- all nodes failed;

Is it because the syntax is not incorrect ? Could it be because of a lack
of memory?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

First, what version of es-hadoop connector are you using?

Your syntax is correct (documentation explaining it here [1]) however
it would help to use es.resource.write instead of just
es.resource. Shouldn't really matter but please try it out. Also
memory should be an issue.

Assuming you are using the latest stable (2.0.x) or dev build, can you
please turn on logging on the rest package [2] and upload the logs
in a gist somewhere? More information here [3]

Cheers,

[1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format
[2] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html
[3] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help

On Tue, Apr 14, 2015 at 6:15 PM, jean.frege62@gmail.com wrote:

Hi,

I have got severals index in my ElasticSearch starting with log-access. I
would like to read all of them thanks to Hive.

I have tried 'es.resource' = 'log-access*/access' which doesn't seem to be
correct.

Then i have tried " 'es.resource' =
'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' =
'log-access/{@timestamp:YYYY.MM.dd}'.
It resulted in the error below even though i have managed reading one index

Connection error (check network and/or proxy settings)- all nodes failed;

Is it because the syntax is not incorrect ? Could it be because of a lack of
memory?

Thanks in advance.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAJogdmc4EjHU3MNiaPgyXSaU8BcJ-eSsYtN46z%2BPM2W8aG98kA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I'm using one of the 4th April 2015 build of elasticsearch-hadoop-2.1.0.

Doesn't 'es.resource.write' only work for writing index? I'm trying to read
several ones.

Does the connector tells user about lack of memory or is the Connection
Error message a strandard message ?

I will take a look at logging possibilities and come back to you.

Le mercredi 15 avril 2015 11:02:43 UTC+2, Costin Leau a écrit :

First, what version of es-hadoop connector are you using?

Your syntax is correct (documentation explaining it here [1]) however
it would help to use es.resource.write instead of just
es.resource. Shouldn't really matter but please try it out. Also
memory should be an issue.

Assuming you are using the latest stable (2.0.x) or dev build, can you
please turn on logging on the rest package [2] and upload the logs
in a gist somewhere? More information here [3]

Cheers,

[1]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format
[2]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html
[3]
http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help

On Tue, Apr 14, 2015 at 6:15 PM, <jean.f...@gmail.com <javascript:>>
wrote:

Hi,

I have got severals index in my ElasticSearch starting with log-access.
I
would like to read all of them thanks to Hive.

I have tried 'es.resource' = 'log-access*/access' which doesn't seem to
be
correct.

Then i have tried " 'es.resource' =
'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' =
'log-access/{@timestamp:YYYY.MM.dd}'.
It resulted in the error below even though i have managed reading one
index

Connection error (check network and/or proxy settings)- all nodes
failed;

Is it because the syntax is not incorrect ? Could it be because of a
lack of
memory?

Thanks in advance.

--
You received this message because you are subscribed to the Google
Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On 4/15/15 3:49 PM, jean.frege62@gmail.com wrote:

I'm using one of the 4th April 2015 build of elasticsearch-hadoop-2.1.0.

Doesn't 'es.resource.write' only work for writing index? I'm trying to read several ones.

Well that's your problem - you are trying to read not to write based on pattern. Which is not possible because how does
the connector know what value to actually use for the pattern?
In case of writing, the pattern is replaced by the values about to be indexed in Elasticsearch. In case of reading
though, there is no value - the value is in Elasticsearch which cannot be accessed because the index refers potentially
to a value contained by it before being resolved in the first place.

Does the connector tells user about lack of memory or is the Connection Error message a strandard message ?

I will take a look at logging possibilities and come back to you.

Le mercredi 15 avril 2015 11:02:43 UTC+2, Costin Leau a écrit :

First, what version of es-hadoop connector are you using?

Your syntax is correct (documentation explaining it here [1]) however
it would help to use `es.resource.write` instead of just
`es.resource`. Shouldn't really matter but please try it out. Also
memory should be an issue.

Assuming you are using the latest stable (2.0.x) or dev build, can you
please turn on logging on the `rest` package [2] and upload the logs
in a gist somewhere? More information here [3]

Cheers,

[1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format
<http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format>
[2] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html
<http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html>
[3] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help
<http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help>

On Tue, Apr 14, 2015 at 6:15 PM,  <jean.f...@gmail.com <javascript:>> wrote:
> Hi,
>
> I have got severals index in my ElasticSearch starting with log-access. I
> would like to read all of them thanks to Hive.
>
> I have tried 'es.resource' = 'log-access*/access' which doesn't seem to be
> correct.
>
> Then i have tried " 'es.resource' =
> 'log-access{@timestamp:YYYY.MM.dd}/access' " and  "'es.resource' =
> 'log-access/{@timestamp:YYYY.MM.dd}'.
> It resulted in the error below even though i have managed reading one index
>
> Connection error (check network and/or proxy settings)- all nodes failed;
>
>
> Is it because the syntax is not incorrect ? Could it be because of a lack of
> memory?
>
> Thanks in advance.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email toelasticsearc...@googlegroups.com <javascript:>.
> To view this discussion on the web visit
>https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com>.
> For more options, visithttps://groups.google.com/d/optout <https://groups.google.com/d/optout>.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/552F5A84.6040409%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

I understand the problem now. I used */type to access all index based on a
type. It did the trick for my problem.

I'm starting to think that including date in my index name wasn't a good
idea.

Le jeudi 16 avril 2015 08:45:48 UTC+2, Costin Leau a écrit :

On 4/15/15 3:49 PM, jean.f...@gmail.com <javascript:> wrote:

I'm using one of the 4th April 2015 build of elasticsearch-hadoop-2.1.0.

Doesn't 'es.resource.write' only work for writing index? I'm trying to
read several ones.

Well that's your problem - you are trying to read not to write based on
pattern. Which is not possible because how does
the connector know what value to actually use for the pattern?
In case of writing, the pattern is replaced by the values about to be
indexed in Elasticsearch. In case of reading
though, there is no value - the value is in Elasticsearch which cannot be
accessed because the index refers potentially
to a value contained by it before being resolved in the first place.

Does the connector tells user about lack of memory or is the Connection
Error message a strandard message ?

I will take a look at logging possibilities and come back to you.

Le mercredi 15 avril 2015 11:02:43 UTC+2, Costin Leau a écrit :

First, what version of es-hadoop connector are you using? 

Your syntax is correct (documentation explaining it here [1]) 

however

it would help to use `es.resource.write` instead of just 
`es.resource`. Shouldn't really matter but please try it out. Also 
memory should be an issue. 

Assuming you are using the latest stable (2.0.x) or dev build, can 

you

please turn on logging on the `rest` package [2] and upload the logs 
in a gist somewhere? More information here [3] 

Cheers, 

[1] 

http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format

<

http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format>

[2] 

http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html

<

http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html>

[3] 

http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help

<

http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help>

On Tue, Apr 14, 2015 at 6:15 PM,  <jean.f...@gmail.com 

<javascript:>> wrote:

> Hi, 
> 
> I have got severals index in my ElasticSearch starting with 

log-access. I

> would like to read all of them thanks to Hive. 
> 
> I have tried 'es.resource' = 'log-access*/access' which doesn't 

seem to be

> correct. 
> 
> Then i have tried " 'es.resource' = 
> 'log-access{@timestamp:YYYY.MM.dd}/access' " and  "'es.resource' = 
> 'log-access/{@timestamp:YYYY.MM.dd}'. 
> It resulted in the error below even though i have managed reading 

one index

> 
> Connection error (check network and/or proxy settings)- all nodes 

failed;

> 
> 
> Is it because the syntax is not incorrect ? Could it be because of 

a lack of

> memory? 
> 
> Thanks in advance. 
> 
> -- 
> You received this message because you are subscribed to the Google 

Groups

> "elasticsearch" group. 
> To unsubscribe from this group and stop receiving emails from it, 

send an

> email toelasticsearc...@googlegroups.com <javascript:>. 
> To view this discussion on the web visit 
>

https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com

<

https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com>.

> For more options, visithttps://groups.google.com/d/optout <

https://groups.google.com/d/optout>.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:> <mailto:
elasticsearch+unsubscribe@googlegroups.com <javascript:>>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com

<
https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com?utm_medium=email&utm_source=footer>.

For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/679d51ca-32ce-489f-8c28-dcabf30f1e99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.