Getting an ElasticSearch Cluster to Green (Cluster Setup on OS X)


(David James) #1

I would appreciate some help here or over on StackOverflow... Thanks! -David

Getting an ElasticSearch Cluster to Green (Cluster Setup on OS X)

I have installed ElasticSearch on Mac OS X using Homebrewhttps://github.com/mxcl/homebrew/blob/master/Library/Formula/elasticsearch.rb.
It works. The cluster started off with "green" healthhttp://www.elasticsearch.org/guide/reference/api/admin-cluster-health.html.
However, right after adding data, it has gone to "yellow".

The cluster health is status is: green, yellow or red. On the shard level,
a red status indicates that the specific shard is not allocated in the
cluster, yellow means that the primary shard is allocated but replicas are
not, and green means that all shards are allocated. The index level status
is controlled by the worst shard status. The cluster status is controlled
by the worst index status.

So, my replica shards are not allocated. How do I allocate them? (I'm
thinking out loud.)

According to Shay on "I keep getting cluster health status of Yellow"http://elasticsearch-users.115913.n3.nabble.com/I-keep-getting-cluster-health-status-of-Yellow-td929520.html:
"the shard allocation mechanism does not allocate a shard and its replica
on the same node, though it does allocate different shards on the same
node. So, you will need two nodes to get cluster state of green."

So, I need to start up a second node. I did this by:

cd ~/Library/LaunchAgents/
cp homebrew.mxcl.elasticsearch.plist homebrew.mxcl.elasticsearch-2.plist
launchctl load -wF ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch-2.plist

Now I have "Korvus" on http://localhost:9200/ and "Iron Monger" on
http://localhost:9201/. Woot. But, I don't see any indications that they
know about each other. How do I connect / introduce them to each other?

Note: I read Zen Discoveryhttp://www.elasticsearch.org/guide/reference/modules/discovery/zen.html,
but do not feel enlightened yet.

--


(David Pilato) #2

If I correctly understood your concern, you started one node, then create an index with default settings and you see a yellow cluster.

If you are just testing ES on your laptop, I suggest to work with yellow cluster or to modify your index settings to have replica=0 (your cluster will be Green).

When you start a second node with default settings, all nodes automagicaly discover each others.
But, you wrote that you copied first es dir to second es dir. Ok. But, you should remove data dir before starting the second node. I don't know if there's any side effect if you also copy data dir...

HTH
David

--

Le 14 août 2012 à 04:23, David James davidcjames@gmail.com a écrit :

I would appreciate some help here or over on StackOverflow... Thanks! -David

Getting an ElasticSearch Cluster to Green (Cluster Setup on OS X)
http://stackoverflow.com/questions/11944915/getting-an-elasticsearch-cluster-to-green

I have installed ElasticSearch on Mac OS X using Homebrew. It works. The cluster started off with "green" health. However, right after adding data, it has gone to "yellow".

The cluster health is status is: green, yellow or red. On the shard level, a red status indicates that the specific shard is not allocated in the cluster, yellow means that the primary shard is allocated but replicas are not, and green means that all shards are allocated. The index level status is controlled by the worst shard status. The cluster status is controlled by the worst index status.

So, my replica shards are not allocated. How do I allocate them? (I'm thinking out loud.)

According to Shay on "I keep getting cluster health status of Yellow": "the shard allocation mechanism does not allocate a shard and its replica on the same node, though it does allocate different shards on the same node. So, you will need two nodes to get cluster state of green."

So, I need to start up a second node. I did this by:

cd ~/Library/LaunchAgents/
cp homebrew.mxcl.elasticsearch.plist homebrew.mxcl.elasticsearch-2.plist
launchctl load -wF ~/Library/LaunchAgents/homebrew.mxcl.elasticsearch-2.plist
Now I have "Korvus" on http://localhost:9200/ and "Iron Monger" onhttp://localhost:9201/. Woot. But, I don't see any indications that they know about each other. How do I connect / introduce them to each other?

Note: I read Zen Discovery, but do not feel enlightened yet.

--

--


(David James) #3

Solved. Do not use network.host: 127.0.0.1 Leave that line commented out
to let it be automatically derived. The default elasticsearch.yml was
correct. A configuration tweak by the Homebrew installer had incorrectly
(IMO) specified the 127.0.0.1 loopback interface.

--


(Karel Minarik) #4

Hi, if the only thing you would be after is a green cluster, then indeed
setting number_of_replicas : 1 is the correct way on a local system;
see https://github.com/elasticsearch/elasticsearch/blob/master/config/elasticsearch.yml#L111-115.

The Homebrew installer tweaks the network.host setting since
elasticsearch would "freeze" when changing networks, getting new IP from
router, etc etc on a laptop. Sounds like it should be improved, though --
thanks for filing the issue with Homebrew.

Karel

P.S.
A personal note: it's not very cool to crosspost the same question on the
mailing list and StackOverflow...

On Tuesday, August 14, 2012 7:01:36 AM UTC+2, David James wrote:

Solved. Do not use network.host: 127.0.0.1 Leave that line commented out
to let it be automatically derived. The default elasticsearch.yml was
correct. A configuration tweak by the Homebrew installer had incorrectly
(IMO) specified the 127.0.0.1 loopback interface.

--


(David James) #5

Says who? I disagree. I didn't hide the cross-post. It isn't cool not
finding the answers you need in either place. (BTW, SO was more helpful
in this case.) I'm helping everyone out by sharing.

A personal note: it's not very cool to crosspost the same question on the

mailing list and StackOverflow...

--


(David Pilato) #6

My opinion on this is I don't like to use SO to get help as it's not described in the official help page: http://www.elasticsearch.org/help/

My 2 cents.
David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 14 août 2012 à 18:14, David James davidcjames@gmail.com a écrit :

Says who? I disagree. I didn't hide the cross-post. It isn't cool not finding the answers you need in either place. (BTW, SO was more helpful in this case.) I'm helping everyone out by sharing.

A personal note: it's not very cool to crosspost the same question on the mailing list and StackOverflow...

--


(David James) #7

Wow. :slight_smile: You better not ask a friend or colleague either, because they are
not listed on that page. :stuck_out_tongue:

Some perspectives:

  1. You can use a user-moderated, editable, living Q&A system that doesn't
    suffer from the "threading" problem -- or you can go with "simple" (IMHO,
    not as good) mailing list technology. There are several good Q&A systems, I
    like StackOverflow myself.
  2. From a "keep it public, archived, and available" POV, I suppose it
    might be argued that it is easier to extract, archive, and transition
    Google Group content than SO content. I don't know -- haven't tried either
    myself. SO content is available under the Creative Commons Attribution
    Share Alike license -- not bad IMO. (See http://stackexchange.com/legal)
  3. From a "commercial" POV, you can support Google (here) or Stack Exchange
    Inc (StackOverflow).

My opinion on this is I don't like to use SO to get help as it's not

described in the official help page: http://www.elasticsearch.org/help/

--


(Clinton Gormley) #8

On Tue, 2012-08-14 at 11:15 -0700, David James wrote:

Wow. :slight_smile: You better not ask a friend or colleague either, because they
are not listed on that page. :stuck_out_tongue:

:slight_smile:

My feeling is that SO doesn't have the strong elasticsearch community,
so by asking your question there, there is a good chance it'll be
exposed to fewer people who really know what they're talking about.

That said, the formatting of SO and the durability of the answers means
that I tend to spend more time answering questions there in greater
detail, because its easy to link people asking the same question to the
right answer in the future. (and its easier to search than google
groups).

clint

--


(Karel Minarik) #9

On Tuesday, August 14, 2012 6:14:23 PM UTC+2, David James wrote:

Says who?

It was clearly marked as a personal note, so, “says me”. At the same time,
I respect your disagreement.

The cross-posting of questions between GitHub issues, StackOverflow and
Google Groups is rather unfortunate, in my personal opinion. My personal
opinion is further supported by taking care of discussions related to the
Tire Ruby client for elasticsearch.

As Clinton Gormley notes in the discussion below, StackOverflow is perfect
for asking and answering higher level, conceptual questions, not cases like
"I have a simple problem, now what".

In the grand scheme, it's a matter of which source of information wins in
the Google fight -- if you're interested in a quick solution to a problem,
the IRC is still hands down the most effective communication channel.

The mailing list is a perfect channel for the type of the question you
originally asked, at the moment.

Karel

--


(David Pilato) #10

You're right. I never ask a colleague ! :wink:
No, just kidding.

Fact is I answer in the ML every day but I don't look at other places (SO). I should do it. I think I will. But by now, IMHO, I think that you will get less answers in SO than here.

But, if Shay and ES team are ok to consider SO as a good place to ask for help, it should be added to the help page.

David

--

Le 14 août 2012 à 20:15, David James davidcjames@gmail.com a écrit :

Wow. :slight_smile: You better not ask a friend or colleague either, because they are not listed on that page. :stuck_out_tongue:

Some perspectives:

  1. You can use a user-moderated, editable, living Q&A system that doesn't suffer from the "threading" problem -- or you can go with "simple" (IMHO, not as good) mailing list technology. There are several good Q&A systems, I like StackOverflow myself.
  2. From a "keep it public, archived, and available" POV, I suppose it might be argued that it is easier to extract, archive, and transition Google Group content than SO content. I don't know -- haven't tried either myself. SO content is available under the Creative Commons Attribution Share Alike license -- not bad IMO. (See http://stackexchange.com/legal)
  3. From a "commercial" POV, you can support Google (here) or Stack Exchange Inc (StackOverflow).

My opinion on this is I don't like to use SO to get help as it's not described in the official help page: http://www.elasticsearch.org/help/

--


(Jörg Prante) #11

I was asked on Stackoverflow about the JDBC river before and I will
continue to encourage everyone to join the Elasticsearch community here to
ask for help.

Scattering help resources for an open-source project hinders coherent
documentation and support. Please keep follow this mailing list, enhance
the docs, write useful recipes, beginner tutorials, cookbooks. Answering
questions on places such as Stackoverflow will have a tendency to drain
energy from the community.

My 2c.

Best regards,

Jörg

On Wednesday, August 15, 2012 12:00:52 AM UTC+2, David Pilato wrote:

You're right. I never ask a colleague ! :wink:
No, just kidding.

Fact is I answer in the ML every day but I don't look at other places
(SO). I should do it. I think I will. But by now, IMHO, I think that you
will get less answers in SO than here.

But, if Shay and ES team are ok to consider SO as a good place to ask for
help, it should be added to the help page.

David

--

Le 14 août 2012 à 20:15, David James <david...@gmail.com <javascript:>> a
écrit :

Wow. :slight_smile: You better not ask a friend or colleague either, because they are
not listed on that page. :stuck_out_tongue:

Some perspectives:

  1. You can use a user-moderated, editable, living Q&A system that doesn't
    suffer from the "threading" problem -- or you can go with "simple" (IMHO,
    not as good) mailing list technology. There are several good Q&A systems, I
    like StackOverflow myself.
  2. From a "keep it public, archived, and available" POV, I suppose it
    might be argued that it is easier to extract, archive, and transition
    Google Group content than SO content. I don't know -- haven't tried either
    myself. SO content is available under the Creative Commons Attribution
    Share Alike license -- not bad IMO. (See http://stackexchange.com/legal)
  3. From a "commercial" POV, you can support Google (here) or Stack
    Exchange Inc (StackOverflow).

My opinion on this is I don't like to use SO to get help as it's not

described in the official help page: http://www.elasticsearch.org/help/

--

--


(David James) #12

Thanks everyone for explaining your thoughts.

A vibrant community can and should engage and solve problems in various
ways, places, and styles. Figuring out the "better" places and way to ask a
question is an art. All channels are welcome. :slight_smile: SO isn't going anywhere.
If we want to support getting ElasticSearch knowledge out there, it is an
important part of the ecosystem.

Karel, what you write about SO "StackOverflow is perfect for asking and
answering higher level, conceptual questions" is interesting, but I may
disagree. SO does not encourage "conceptual" questions:

"You should only ask practical, answerable questions based on actualproblems that you face.

Chatty, open-ended questions diminish the usefulness of our site
and push other questions off the front page. ... Your questions should be
reasonably scoped. ... If your motivation for asking the question is “I
would like to participate in a discussion about ______”, then you should
not be asking here." (from http://stackoverflow.com/faq)

I'll continue to share across various channels and forums. I tend to find
that mailing lists still work better than SO for arbitrary conversations
that don'f fit the Q&A pattern.

--


(system) #13