Elastic Search is always status red when I restart with no data in index


(Sam Millman) #1

This topic is heavily linked to:

Using the data there, the exact log I kept getting red status whenever I
restarted. eventually through the effort of continuous trial and error I
discovered the problem came down to the fact that I created my indexes and
mappings and then SIGTERM killed the node without any data on it.

For some reason Elastic Search thinks that it cannot find all the documents
when in fact there are no documents to be found.

Is this a bug or am I just being stupid?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/78f66878-a20d-4b4f-adf7-6f091c1b0d38%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jörg Prante) #2

Can you provide your logs for review somewhere, and the exact sequence of
commands you executed? Thanks.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHoLyz8JFRzB%3DNn14yTLkcHDLrz5jZ2WHrw%3D3OVMFocUw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Sam Millman) #3

That is the weird thing, what you see in that SO question is my logs, it
starts with the first node and then I restart, there are no errors or
warnings, nothing really.

The only way I know something is wrong is because it gives me the 503 when
I come to query and of course HQ plugin says the status is red.

Other than that there is no indication from elastic itself that anything is
wrong.

The exact commands I executed where:

sam@ubuntu:~/elasticsearch-0.90.8/bin$ sudo ./elasticsearch

After having downloaded the source, then I run my index setup script:

<?php function p($message){ print $message . "\n"; } require '../glue/components/vendor/autoload.php'; $client = new Elasticsearch\Client(); $init = false; p("OK! Setting up Elastic Search"); if($client->indices()->exists(array('index' => 'main')) && in_array('--reindex', $_SERVER['argv'])){ p("Deleting the index..."); var_dump($client->indices()->delete(array('index' => 'main'))); p("Sleeping 10 seconds to let the delete take effect"); sleep(10); } if(!$client->indices()->exists(array('index' => 'main')) || in_array('--reindex', $_SERVER['argv'])){ p("Creating the index..."); var_dump($client->indices()->create(array( 'index' => 'main', 'body' => array( 'settings' => array( 'number_of_shards' => 5, 'number_of_replicas' => 2, 'analysis' => array( 'analyzer' => array( 'noStopFilter' => array( 'tokenizer' => 'standard', 'filter' => array("standard", "lowercase"), ) ) ) ) )))); $init = true; p("Sleeping 10 seconds to let the index take effect"); sleep(10); } if(in_array('--reindex', $_SERVER['argv']) || in_array('--remap', $_SERVER['argv']) || $init){ p('Adding the mapping...'); var_dump($client->indices()->putMapping(array( 'index' => 'main', 'type' => '_default_', 'body' => array( '_default_' => array( 'properties' => array( 'title' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ), 'userId' => array( 'type' => 'string', 'index' => 'not_analyzed' ), 'username' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ), 'tags' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ) ) ) ) ))); var_dump($client->indices()->putMapping(array( 'index' => 'main', 'type' => 'help', 'body' => array( 'help' => array( 'properties' => array( 'title' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ), 'normalisedTitle' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ), 'path' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ), 'username' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ), 'tags' => array( 'type' => 'string', 'analyzer' => 'noStopFilter' ) ) ) ) ))); p('done adding the mapping'); } p('done setting up Elastic Search...let\'s see if it works'); And then I search for a program on the port: sudo netstat -lpn |grep :9200 And simply kill it: sudo kill 9567 with 9567 being the process id. and then rerun the first command: sam@ubuntu:~/elasticsearch-0.90.8/bin$ sudo ./elasticsearch And that is when I get status red. On Sunday, 22 December 2013 17:00:01 UTC, Jörg Prante wrote: > > Can you provide your logs for review somewhere, and the exact sequence of > commands you executed? Thanks. > > Jörg > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/60bd35f3-1629-4d39-95bb-c5e73cc4d421%40googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.

(Jörg Prante) #4

You have a single node, and number_of_replicas = 2. That will not work. It
means, "create an index, but each shard should exist three times. And, If
recovery is required, wait for a quorum of shards (3+1)/2 = 2 for recovery
to begin, before setting the state to yellow".

Why is it wrong? Well, you have one node, and therefore, at maximum a
quorum 1 of shards can be present. Creating replica on a single node is
quite useless. Worse, increasing replica level from the default of 1 to 2
tells a single node ES cluster not to recover at all, but instead wait for
more nodes that can fulfill the quorum of shards.

Use

gateway.local.initial_shards: 1

in the config to overwrite the default value of "quorum", and the gateway
will recover the index.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOwxyJ6PgjFkq9%3Dz47o%2B_kieayK-L09Ur9T9FGVrHeTw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


RED status in elastic search indices
(Sam Millman) #5

Ok that might explain more actually because when I enter data is actually
create 2 nodes so that would explain why it can resolve the quorum.

I initially put those settings in because that's what I intend to use in
production but I'll try that config setting out in a sec thanks :).

On Sunday, 22 December 2013 18:49:10 UTC, Jörg Prante wrote:

You have a single node, and number_of_replicas = 2. That will not work. It
means, "create an index, but each shard should exist three times. And, If
recovery is required, wait for a quorum of shards (3+1)/2 = 2 for recovery
to begin, before setting the state to yellow".

Why is it wrong? Well, you have one node, and therefore, at maximum a
quorum 1 of shards can be present. Creating replica on a single node is
quite useless. Worse, increasing replica level from the default of 1 to 2
tells a single node ES cluster not to recover at all, but instead wait for
more nodes that can fulfill the quorum of shards.

Use

gateway.local.initial_shards: 1

in the config to overwrite the default value of "quorum", and the gateway
will recover the index.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5ff77f31-126d-4f34-92b0-e8a344468a5a%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #6

You'll also probably find it easier to use the .deb package for ubuntu, as
it uses an init script and will create logs in /var/log.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 December 2013 05:56, Sam Millman sam.millman@gmail.com wrote:

Ok that might explain more actually because when I enter data is actually
create 2 nodes so that would explain why it can resolve the quorum.

I initially put those settings in because that's what I intend to use in
production but I'll try that config setting out in a sec thanks :).

On Sunday, 22 December 2013 18:49:10 UTC, Jörg Prante wrote:

You have a single node, and number_of_replicas = 2. That will not work.
It means, "create an index, but each shard should exist three times. And,
If recovery is required, wait for a quorum of shards (3+1)/2 = 2 for
recovery to begin, before setting the state to yellow".

Why is it wrong? Well, you have one node, and therefore, at maximum a
quorum 1 of shards can be present. Creating replica on a single node is
quite useless. Worse, increasing replica level from the default of 1 to 2
tells a single node ES cluster not to recover at all, but instead wait for
more nodes that can fulfill the quorum of shards.

Use

gateway.local.initial_shards: 1

in the config to overwrite the default value of "quorum", and the gateway
will recover the index.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5ff77f31-126d-4f34-92b0-e8a344468a5a%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZCGeuR9_DSMuMEGqavyh7nsXpOqAe0qGxEQYxfR01JWw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Sam Millman) #7

True that, though the normal download does the same but in the root of the
elastic folder.

When I deploy I will most likely use the deb.

Thanks,

On Sunday, 22 December 2013 22:03:42 UTC, Mark Walkom wrote:

You'll also probably find it easier to use the .deb package for ubuntu, as
it uses an init script and will create logs in /var/log.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 23 December 2013 05:56, Sam Millman <sam.m...@gmail.com <javascript:>>wrote:

Ok that might explain more actually because when I enter data is actually
create 2 nodes so that would explain why it can resolve the quorum.

I initially put those settings in because that's what I intend to use in
production but I'll try that config setting out in a sec thanks :).

On Sunday, 22 December 2013 18:49:10 UTC, Jörg Prante wrote:

You have a single node, and number_of_replicas = 2. That will not work.
It means, "create an index, but each shard should exist three times. And,
If recovery is required, wait for a quorum of shards (3+1)/2 = 2 for
recovery to begin, before setting the state to yellow".

Why is it wrong? Well, you have one node, and therefore, at maximum a
quorum 1 of shards can be present. Creating replica on a single node is
quite useless. Worse, increasing replica level from the default of 1 to 2
tells a single node ES cluster not to recover at all, but instead wait for
more nodes that can fulfill the quorum of shards.

Use

gateway.local.initial_shards: 1

in the config to overwrite the default value of "quorum", and the
gateway will recover the index.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/5ff77f31-126d-4f34-92b0-e8a344468a5a%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/114dbb0c-cbbe-4274-913f-69fe0d6a3fa6%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8