Elasticsearch + Wonderdog + Pig | Master Not Discovered

Hi,

I'm trying to play around with pig and elasticsearch locally on my mac. I
came across this posthttp://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/that details the steps required to pass data from an avro file into
elasticsearch.

When I run the pig script locally (pig -l /tmp -x local -v -w myscript.pig)
I get a warning
WARN org.elasticsearch.discovery - [Meteorite] waited for 30s and no
initial state was set by the discovery
and then an error
java.lang.RuntimeException:
org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
which seem to centre around discovery problems.

Following advice elsewherehttp://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.htmlI've tried adding the following to my elastic search.yml configuation:
network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false

But that didn't seem to help. What else should I be checking?

Thanks in advance,
Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

what's the cluster name of your ES cluster? IIRC, last time I played
around with wonderdog it used the default cluster name of 'elasticsearch',
that bit me for a bit, there was no way to configure Wonderdog (through
Pig) to use a different cluster name.

Paul

On 23 May 2013 22:01, Tim Bowker tbowker@gmail.com wrote:

Hi,

I'm trying to play around with pig and elasticsearch locally on my mac. I
came across this posthttp://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/that details the steps required to pass data from an avro file into
elasticsearch.

When I run the pig script locally (pig -l /tmp -x local -v -w
myscript.pig) I get a warning
WARN org.elasticsearch.discovery - [Meteorite] waited for 30s and no
initial state was set by the discovery
and then an error
java.lang.RuntimeException:
org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
which seem to centre around discovery problems.

Following advice elsewherehttp://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.htmlI've tried adding the following to my elastic search.yml configuation:
network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false

But that didn't seem to help. What else should I be checking?

Thanks in advance,
Tim

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I think you should try to add the;

discovery.zen.unicast.hosts: ["youserverip:9300"] if you disabled multicast.

Moreover, as mentionned you should check if you have set the right cluster
name in the config.

On Thursday, May 23, 2013 8:01:34 AM UTC-4, Tim Bowker wrote:

Hi,

I'm trying to play around with pig and elasticsearch locally on my mac. I
came across this posthttp://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/that details the steps required to pass data from an avro file into
elasticsearch.

When I run the pig script locally (pig -l /tmp -x local -v -w
myscript.pig) I get a warning
WARN org.elasticsearch.discovery - [Meteorite] waited for 30s and no
initial state was set by the discovery
and then an error
java.lang.RuntimeException:
org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
which seem to centre around discovery problems.

Following advice elsewherehttp://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.htmlI've tried adding the following to my elastic search.yml configuation:
network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false

But that didn't seem to help. What else should I be checking?

Thanks in advance,
Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

As an alternative you can use elasticsearc-hadoop [1] to read/write data to ES from Hadoop/Pig/Cascading/Hive. Coming
back to your post, you can read data into the AvroStorage and then use ESStorage [2] to push it to ES.
Note that elasticsearch-hadoop uses REST and is self contained (~100Kb), doesn't require the ES client nor to be on the
same machines (though ES collocates nicely with Hadoop).

Hope this helps,

[1] https://github.com/elasticsearch/elasticsearch-hadoop/
[2] https://github.com/elasticsearch/elasticsearch-hadoop/#pig

On 5/23/2013 3:26 PM, Paul Smith wrote:

what's the cluster name of your ES cluster? IIRC, last time I played around with wonderdog it used the default cluster
name of 'elasticsearch', that bit me for a bit, there was no way to configure Wonderdog (through Pig) to use a different
cluster name.

Paul

On 23 May 2013 22:01, Tim Bowker <tbowker@gmail.com mailto:tbowker@gmail.com> wrote:

Hi,

I'm trying to play around with pig and elasticsearch locally on my mac. I came across this post
<http://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/> that
details the steps required to pass data from an avro file into elasticsearch.

When I run the pig script locally (pig -l /tmp -x local -v -w myscript.pig) I get a warning
WARN  org.elasticsearch.discovery - [Meteorite] waited for 30s and no initial state was set by the discovery
and then an error
java.lang.RuntimeException: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
which seem to centre around discovery problems.

Following advice elsewhere
<http://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.html> I've tried adding the
following to my elastic search.yml configuation:
network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false

But that didn't seem to help. What else should I be checking?

Thanks in advance,
Tim

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com <mailto:elasticsearch%2Bunsubscribe@googlegroups.com>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks for all your help. However, I think I'm getting completely out of my
depth... But here goes anyway:

In response to "What's the cluster name of your ES cluster?" and
"add discovery.zen.unicast.hosts: ["youserverip:9300"] if you disabled
multicast."
I updated my config file as follows:

cluster.name: elasticsearch
network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false
discovery.zen.unicast.hosts: ["127.0.0.1:9300"]

But still no luck :frowning:

The library provided by Costin looks interesting. I had a quick go at
getting some data I have manually entered (when I followed the 5 min
guide). My curl search:
curl 'http://localhost:9200/blog/post/_search?q=user:dilbert'
works but when I run my pig script:
A = LOAD 'blog/post/_search?q=user:dilbert' USING
org.elasticsearch.hadoop.pig.ESStorage();
DUMP A;
Unfortunately that fails as well:
2013-05-23 17:17:53,101 [JobControl] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:tbowker
cause:org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
[GET] on [blog/post/_search_shards] failed; server[http://localhost:9200]
returned
[{"_index":"blog","_type":"post","_id":"_search_shards","exists":false}]

Tim

On Thursday, 23 May 2013 14:11:01 UTC+1, Costin Leau wrote:

Hi,

As an alternative you can use elasticsearc-hadoop [1] to read/write data
to ES from Hadoop/Pig/Cascading/Hive. Coming
back to your post, you can read data into the AvroStorage and then use
ESStorage [2] to push it to ES.
Note that elasticsearch-hadoop uses REST and is self contained (~100Kb),
doesn't require the ES client nor to be on the
same machines (though ES collocates nicely with Hadoop).

Hope this helps,

[1] https://github.com/elasticsearch/elasticsearch-hadoop/
[2] https://github.com/elasticsearch/elasticsearch-hadoop/#pig

On 5/23/2013 3:26 PM, Paul Smith wrote:

what's the cluster name of your ES cluster? IIRC, last time I played
around with wonderdog it used the default cluster
name of 'elasticsearch', that bit me for a bit, there was no way to
configure Wonderdog (through Pig) to use a different
cluster name.

Paul

On 23 May 2013 22:01, Tim Bowker <tbo...@gmail.com <javascript:><mailto:
tbo...@gmail.com <javascript:>>> wrote:

Hi, 

I'm trying to play around with pig and elasticsearch locally on my 

mac. I came across this post

<

http://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/>
that

details the steps required to pass data from an avro file into 

elasticsearch.

When I run the pig script locally (pig -l /tmp -x local -v -w 

myscript.pig) I get a warning

WARN  org.elasticsearch.discovery - [Meteorite] waited for 30s and 

no initial state was set by the discovery

and then an error 
java.lang.RuntimeException: 

org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

which seem to centre around discovery problems. 

Following advice elsewhere 
<

http://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.html>
I've tried adding the

following to my elastic search.yml configuation: 
network.host: 127.0.0.1 
discovery.zen.ping.multicast.enabled: false 

But that didn't seem to help. What else should I be checking? 

Thanks in advance, 
Tim 

-- 
You received this message because you are subscribed to the Google 

Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, 

send an email to

elasticsearc...@googlegroups.com <javascript:> <mailto:

elasticsearch%2Bunsubscribe@googlegroups.com <javascript:>>.

For more options, visit https://groups.google.com/groups/opt_out. 

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

discovery.zen.unicast.hosts: ["127.0.0.1:9300"] <-- this should be your
Elasticsearch server IP! Because using it that way, it will try to connect
to the hadoop node which is not what you want.

On Thursday, May 23, 2013 12:23:53 PM UTC-4, Tim Bowker wrote:

Thanks for all your help. However, I think I'm getting completely out of
my depth... But here goes anyway:

In response to "What's the cluster name of your ES cluster?" and
"add discovery.zen.unicast.hosts: ["youserverip:9300"] if you disabled
multicast."
I updated my config file as follows:

cluster.name: elasticsearch
network.host: 127.0.0.1
discovery.zen.ping.multicast.enabled: false
discovery.zen.unicast.hosts: ["127.0.0.1:9300"]

But still no luck :frowning:

The library provided by Costin looks interesting. I had a quick go at
getting some data I have manually entered (when I followed the 5 min
guide). My curl search:
curl 'http://localhost:9200/blog/post/_search?q=user:dilbert'
works but when I run my pig script:
A = LOAD 'blog/post/_search?q=user:dilbert' USING
org.elasticsearch.hadoop.pig.ESStorage();
DUMP A;
Unfortunately that fails as well:
2013-05-23 17:17:53,101 [JobControl] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:tbowker
cause:org.apache.pig.backend.executionengine.ExecException: ERROR 2118:
[GET] on [blog/post/_search_shards] failed; server[http://localhost:9200]
returned
[{"_index":"blog","_type":"post","_id":"_search_shards","exists":false}]

Tim

On Thursday, 23 May 2013 14:11:01 UTC+1, Costin Leau wrote:

Hi,

As an alternative you can use elasticsearc-hadoop [1] to read/write data
to ES from Hadoop/Pig/Cascading/Hive. Coming
back to your post, you can read data into the AvroStorage and then use
ESStorage [2] to push it to ES.
Note that elasticsearch-hadoop uses REST and is self contained (~100Kb),
doesn't require the ES client nor to be on the
same machines (though ES collocates nicely with Hadoop).

Hope this helps,

[1] https://github.com/elasticsearch/elasticsearch-hadoop/
[2] https://github.com/elasticsearch/elasticsearch-hadoop/#pig

On 5/23/2013 3:26 PM, Paul Smith wrote:

what's the cluster name of your ES cluster? IIRC, last time I played
around with wonderdog it used the default cluster
name of 'elasticsearch', that bit me for a bit, there was no way to
configure Wonderdog (through Pig) to use a different
cluster name.

Paul

On 23 May 2013 22:01, Tim Bowker <tbo...@gmail.com <mailto:
tbo...@gmail.com>> wrote:

Hi, 

I'm trying to play around with pig and elasticsearch locally on my 

mac. I came across this post

<

http://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/>
that

details the steps required to pass data from an avro file into 

elasticsearch.

When I run the pig script locally (pig -l /tmp -x local -v -w 

myscript.pig) I get a warning

WARN  org.elasticsearch.discovery - [Meteorite] waited for 30s and 

no initial state was set by the discovery

and then an error 
java.lang.RuntimeException: 

org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]

which seem to centre around discovery problems. 

Following advice elsewhere 
<

http://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.html>
I've tried adding the

following to my elastic search.yml configuation: 
network.host: 127.0.0.1 
discovery.zen.ping.multicast.enabled: false 

But that didn't seem to help. What else should I be checking? 

Thanks in advance, 
Tim 

-- 
You received this message because you are subscribed to the Google 

Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, 

send an email to

elasticsearc...@googlegroups.com <mailto:

elasticsearch%2Bunsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/groups/opt_out. 

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to
elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Make sure you're using Elasticsearch version 0.90 or higher - es-hadoop uses a feature in 0.90 to 'split' the queries
across the shards.

On 5/23/2013 7:23 PM, Tim Bowker wrote:

The library provided by Costin looks interesting. I had a quick go at getting some data I have manually entered (when I
followed the 5 min guide). My curl search:
curl 'http://localhost:9200/blog/post/_search?q=user:dilbert'
works but when I run my pig script:
A = LOAD 'blog/post/_search?q=user:dilbert' USING org.elasticsearch.hadoop.pig.ESStorage();
DUMP A;
Unfortunately that fails as well:
2013-05-23 17:17:53,101 [JobControl] ERROR org.apache.hadoop.security.UserGroupInformation - PriviledgedActionException
as:tbowker cause:org.apache.pig.backend.executionengine.ExecException: ERROR 2118: [GET] on [blog/post/_search_shards]
failed; server[http://localhost:9200] returned [{"_index":"blog","_type":"post","_id":"_search_shards","exists":false}]

Tim

On Thursday, 23 May 2013 14:11:01 UTC+1, Costin Leau wrote:

Hi,

As an alternative you can use elasticsearc-hadoop [1] to read/write data to ES from Hadoop/Pig/Cascading/Hive. Coming
back to your post, you can read data into the AvroStorage and then use ESStorage [2] to push it to ES.
Note that elasticsearch-hadoop uses REST and is self contained (~100Kb), doesn't require the ES client nor to be on the
same machines (though ES collocates nicely with Hadoop).

Hope this helps,

[1] https://github.com/elasticsearch/elasticsearch-hadoop/ <https://github.com/elasticsearch/elasticsearch-hadoop/>
[2] https://github.com/elasticsearch/elasticsearch-hadoop/#pig
<https://github.com/elasticsearch/elasticsearch-hadoop/#pig>

On 5/23/2013 3:26 PM, Paul Smith wrote:
> what's the cluster name of your ES cluster?  IIRC, last time I played around with wonderdog it used the default cluster
> name of 'elasticsearch', that bit me for a bit, there was no way to configure Wonderdog (through Pig) to use a different
> cluster name.
>
> Paul
>
>
> On 23 May 2013 22:01, Tim Bowker <tbo...@gmail.com <javascript:> <mailto:tbo...@gmail.com <javascript:>>> wrote:
>
>     Hi,
>
>     I'm trying to play around with pig and elasticsearch locally on my mac. I came across this post
>     <http://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/
<http://hortonworks.com/blog/search-data-at-scale-in-five-minutes-with-pig-wonderdog-and-elasticsearch/>> that
>     details the steps required to pass data from an avro file into elasticsearch.
>
>     When I run the pig script locally (pig -l /tmp -x local -v -w myscript.pig) I get a warning
>     WARN  org.elasticsearch.discovery - [Meteorite] waited for 30s and no initial state was set by the discovery
>     and then an error
>     java.lang.RuntimeException: org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [30s]
>     which seem to centre around discovery problems.
>
>     Following advice elsewhere
>     <http://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.html
<http://elasticsearch-users.115913.n3.nabble.com/MasterNotDiscoveredException-td3940481.html>> I've tried adding the
>     following to my elastic search.yml configuation:
>     network.host: 127.0.0.1
>     discovery.zen.ping.multicast.enabled: false
>
>     But that didn't seem to help. What else should I be checking?
>
>     Thanks in advance,
>     Tim
>
>     --
>     You received this message because you are subscribed to the Google Groups "elasticsearch" group.
>     To unsubscribe from this group and stop receiving emails from it, send an email to
>elasticsearc...@googlegroups.com <javascript:> <mailto:elasticsearch%2Bunsubscribe@googlegroups.com <javascript:>>.
>     For more options, visithttps://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
>elasticsearc...@googlegroups.com <javascript:>.
> For more options, visithttps://groups.google.com/groups/opt_out <https://groups.google.com/groups/opt_out>.
>
>

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.