# \[hadoop\] Reading severals index with Hive **URL:** \ **Category:** Elasticsearch\ **Created:** [April 14, 2015, 3:15pm UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233 "2015-04-14T15:15:05Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jean\_frege62](https://avatars.discourse-cdn.com/v4/letter/j/8dc957/32.png) [@jean\_frege62](https://discuss.elastic.co/u/jean_frege62)\ **Post date:** [April 14, 2015, 3:15pm UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233/1 "2015-04-14T15:15:05Z")
Hi, I have got severals index in my ElasticSearch starting with log-access. I would like to read all of them thanks to Hive. I have tried 'es.resource' = 'log-access\*/access' which doesn't seem to be correct. Then i have tried " 'es.resource' = 'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' = 'log-access/{@timestamp:YYYY.MM.dd}'. It resulted in the error below even though i have managed reading one index Connection error (check network and/or proxy settings)- all nodes failed; Is it because the syntax is not incorrect ? Could it be because of a lack of memory? Thanks in advance. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [April 15, 2015, 9:02am UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233/2 "2015-04-15T09:02:03Z")
First, what version of es-hadoop connector are you using? Your syntax is correct (documentation explaining it here [1]) however it would help to use `es.resource.write` instead of just `es.resource`. Shouldn't really matter but please try it out. Also memory should be an issue. Assuming you are using the latest stable (2.0.x) or dev build, can you please turn on logging on the `rest` package [2] and upload the logs in a gist somewhere? More information here [3] Cheers, [1] [Configuration | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format) [2] [Logging | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html) [3] [Troubleshooting | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help) On Tue, Apr 14, 2015 at 6:15 PM, [jean.frege62@gmail.com](mailto:jean.frege62@gmail.com) wrote: > Hi, > > I have got severals index in my Elasticsearch starting with log-access. I > would like to read all of them thanks to Hive. > > I have tried 'es.resource' = 'log-access\*/access' which doesn't seem to be > correct. > > Then i have tried " 'es.resource' = > 'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' = > 'log-access/{@timestamp:YYYY.MM.dd}'. > It resulted in the error below even though i have managed reading one index > > Connection error (check network and/or proxy settings)- all nodes failed; > > Is it because the syntax is not incorrect ? Could it be because of a lack of > memory? > > Thanks in advance. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJogdmc4EjHU3MNiaPgyXSaU8BcJ-eSsYtN46z%2BPM2W8aG98kA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJogdmc4EjHU3MNiaPgyXSaU8BcJ-eSsYtN46z%2BPM2W8aG98kA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jean\_frege62](https://avatars.discourse-cdn.com/v4/letter/j/8dc957/32.png) [@jean\_frege62](https://discuss.elastic.co/u/jean_frege62)\ **Post date:** [April 15, 2015, 12:49pm UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233/3 "2015-04-15T12:49:25Z")
I'm using one of the 4th April 2015 build of elasticsearch-hadoop-2.1.0. Doesn't 'es.resource.write' only work for writing index? I'm trying to read several ones. Does the connector tells user about lack of memory or is the Connection Error message a strandard message ? I will take a look at logging possibilities and come back to you. Le mercredi 15 avril 2015 11:02:43 UTC+2, Costin Leau a écrit : > First, what version of es-hadoop connector are you using? > > Your syntax is correct (documentation explaining it here [1]) however > it would help to use `es.resource.write` instead of just > `es.resource`. Shouldn't really matter but please try it out. Also > memory should be an issue. > > Assuming you are using the latest stable (2.0.x) or dev build, can you > please turn on logging on the `rest` package [2] and upload the logs > in a gist somewhere? More information here [3] > > Cheers, > > [1] > [Configuration | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format) > [2] > [Logging | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html) > [3] > [Troubleshooting | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help) > > On Tue, Apr 14, 2015 at 6:15 PM, \<[jean.f...@gmail.com](mailto:jean.f...@gmail.com) \\> > wrote: > > > Hi, > > > > I have got severals index in my Elasticsearch starting with log-access. > > I > > would like to read all of them thanks to Hive. > > > > I have tried 'es.resource' = 'log-access\*/access' which doesn't seem to > > be > > correct. > > > > Then i have tried " 'es.resource' = > > 'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' = > > 'log-access/{@timestamp:YYYY.MM.dd}'. > > It resulted in the error below even though i have managed reading one > > index > > > > Connection error (check network and/or proxy settings)- all nodes > > failed; > > > > Is it because the syntax is not incorrect ? Could it be because of a > > lack of > > memory? > > > > Thanks in advance. > > > > -- > > You received this message because you are subscribed to the Google > > Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![costin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/costin/32/44950_2.png) [@costin](https://discuss.elastic.co/u/costin)\ **Post date:** [April 16, 2015, 6:45am UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233/4 "2015-04-16T06:45:24Z")
On 4/15/15 3:49 PM, [jean.frege62@gmail.com](mailto:jean.frege62@gmail.com) wrote: > I'm using one of the 4th April 2015 build of elasticsearch-hadoop-2.1.0. > > Doesn't 'es.resource.write' only work for writing index? I'm trying to read several ones. Well that's your problem - you are trying to read not to write based on pattern. Which is not possible because how does the connector know what value to actually use for the pattern? In case of writing, the pattern is replaced by the values about to be indexed in Elasticsearch. In case of reading though, there is no value - the value is in Elasticsearch which cannot be accessed because the index refers potentially to a value contained by it before being resolved in the first place. > Does the connector tells user about lack of memory or is the Connection Error message a strandard message ? > > I will take a look at logging possibilities and come back to you. > > Le mercredi 15 avril 2015 11:02:43 UTC+2, Costin Leau a écrit : > > ``` > First, what version of es-hadoop connector are you using? > > Your syntax is correct (documentation explaining it here [1]) however > it would help to use `es.resource.write` instead of just > `es.resource`. Shouldn't really matter but please try it out. Also > memory should be an issue. > > Assuming you are using the latest stable (2.0.x) or dev build, can you > please turn on logging on the `rest` package [2] and upload the logs > in a gist somewhere? More information here [3] > > Cheers, > > [1] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format > > [2] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html > > [3] http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help > > > On Tue, Apr 14, 2015 at 6:15 PM, > wrote: > > Hi, > > > > I have got severals index in my ElasticSearch starting with log-access. I > > would like to read all of them thanks to Hive. > > > > I have tried 'es.resource' = 'log-access*/access' which doesn't seem to be > > correct. > > > > Then i have tried " 'es.resource' = > > 'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' = > > 'log-access/{@timestamp:YYYY.MM.dd}'. > > It resulted in the error below even though i have managed reading one index > > > > Connection error (check network and/or proxy settings)- all nodes failed; > > > > > > Is it because the syntax is not incorrect ? Could it be because of a lack of > > memory? > > > > Thanks in advance. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email toelasticsearc...@googlegroups.com . > > To view this discussion on the web visit > >https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com > . > > For more options, visithttps://groups.google.com/d/optout . > > ``` > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/552F5A84.6040409%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/552F5A84.6040409%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jean\_frege62](https://avatars.discourse-cdn.com/v4/letter/j/8dc957/32.png) [@jean\_frege62](https://discuss.elastic.co/u/jean_frege62)\ **Post date:** [April 17, 2015, 1:27pm UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233/5 "2015-04-17T13:27:37Z")
I understand the problem now. I used \*/type to access all index based on a type. It did the trick for my problem. I'm starting to think that including date in my index name wasn't a good idea. Le jeudi 16 avril 2015 08:45:48 UTC+2, Costin Leau a écrit : > On 4/15/15 3:49 PM, [jean.f...@gmail.com](mailto:jean.f...@gmail.com) \ wrote: > > > I'm using one of the 4th April 2015 build of elasticsearch-hadoop-2.1.0. > > > > Doesn't 'es.resource.write' only work for writing index? I'm trying to > > read several ones. > > Well that's your problem - you are trying to read not to write based on > pattern. Which is not possible because how does > the connector know what value to actually use for the pattern? > In case of writing, the pattern is replaced by the values about to be > indexed in Elasticsearch. In case of reading > though, there is no value - the value is in Elasticsearch which cannot be > accessed because the index refers potentially > to a value contained by it before being resolved in the first place. > > > Does the connector tells user about lack of memory or is the Connection > > Error message a strandard message ? > > > > I will take a look at logging possibilities and come back to you. > > > > Le mercredi 15 avril 2015 11:02:43 UTC+2, Costin Leau a écrit : > > > > ``` > > First, what version of es-hadoop connector are you using? > > > > Your syntax is correct (documentation explaining it here [1]) > > > > ``` > > however > > > ``` > > it would help to use `es.resource.write` instead of just > > `es.resource`. Shouldn't really matter but please try it out. Also > > memory should be an issue. > > > > Assuming you are using the latest stable (2.0.x) or dev build, can > > > > ``` > > you > > > ``` > > please turn on logging on the `rest` package [2] and upload the logs > > in a gist somewhere? More information here [3] > > > > Cheers, > > > > [1] > > > > ``` > > [Configuration | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format) > > > ``` > > < > > > > ``` > > [Configuration | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/configuration.html#cfg-multi-writes-format)\> > > > ``` > > [2] > > > > ``` > > [Logging | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html) > > > ``` > > < > > > > ``` > > [Logging | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/logging.html)\> > > > ``` > > [3] > > > > ``` > > [Troubleshooting | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help) > > > ``` > > < > > > > ``` > > [Troubleshooting | Elasticsearch for Apache Hadoop [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/hadoop/master/troubleshooting.html#help)\> > > > ``` > > On Tue, Apr 14, 2015 at 6:15 PM, > > > ``` > > \\> wrote: > > > ``` > > > Hi, > > > > > > I have got severals index in my ElasticSearch starting with > > > > ``` > > log-access. I > > > ``` > > > would like to read all of them thanks to Hive. > > > > > > I have tried 'es.resource' = 'log-access*/access' which doesn't > > > > ``` > > seem to be > > > ``` > > > correct. > > > > > > Then i have tried " 'es.resource' = > > > 'log-access{@timestamp:YYYY.MM.dd}/access' " and "'es.resource' = > > > 'log-access/{@timestamp:YYYY.MM.dd}'. > > > It resulted in the error below even though i have managed reading > > > > ``` > > one index > > > ``` > > > > > > Connection error (check network and/or proxy settings)- all nodes > > > > ``` > > failed; > > > ``` > > > > > > > > > Is it because the syntax is not incorrect ? Could it be because of > > > > ``` > > a lack of > > > ``` > > > memory? > > > > > > Thanks in advance. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > > ``` > > Groups > > > ``` > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, > > > > ``` > > send an > > > ``` > > > email toelasticsearc...@googlegroups.com . > > > To view this discussion on the web visit > > > > > > > ``` > > [https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com) > > > ``` > > < > > > > ``` > > [https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7de76101-b5bd-4b2e-ace2-7a81204ceb72%40googlegroups.com)\>. > > > ``` > > > For more options, visithttps://groups.google.com/d/optout < > > > > ``` > > [https://groups.google.com/d/optout](https://groups.google.com/d/optout)\>. > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \ \ > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) \\>. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com) > > > \< > > [https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ce9efbc5-5fbc-4a8b-adfb-59084c951779%40googlegroups.com?utm_medium=email&utm_source=footer)\>. > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > Costin -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/679d51ca-32ce-489f-8c28-dcabf30f1e99%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/679d51ca-32ce-489f-8c28-dcabf30f1e99%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:19am UTC](https://discuss.elastic.co/t/hadoop-reading-severals-index-with-hive/23233/6 "2017-07-06T00:19:02Z")