Number of replicas automatically reset to 1

Elastic In Your Native Tongue Discussions en français
1

Hi,
I got a problem but don't know how to solve it.
I have installed elasticsearch 5 on my server. And created manually my index 'coloc'. It is a single node cluster so i set the number of replicas to 0.
But when i launch my automated task to add data, it works for fifteen minutes and then change status of cluster to yellow. So i get NoServerResponseException. After testing, I discover that the number of replica of my 'coloc' index was reset to 1 ! Why ?

I don't know how to fix it. Any leads to help ?

Thanks

2

Maybe, I have a clue for you.
In my php script, i was not setting the number of replicas. So, question : if i don't say to my my script i don't want replicas, is it setting number_of_replicas to 1 by default ?

Trying to set my settings number_of_replicas to 0 in my php script. Got this error ...

Elasticsearch\Common\Exceptions\Missing404Exception
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"coloc","index_uuid":"na","index":"coloc"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"coloc","index_uuid":"na","index":"coloc"},"status":404}

ADD : Even if I add a number_of_replicas setting in my script, it resets to 1, then the cluster is yellow and the script fails....

3

Not sure what "your script" is.
But have a look at index templates.

4

thanks, where i can find that ? Even if i don't launch the script (only made to pass data to ES) the status changes to yellow without reasons ...
Add : to be precise:
i create my index 'coloc', set the number_of_replicas to 0 then fill my index with a script. ES create automatic template to fill an index ? If you need my script tell me. I really need help on this :wink:

5

If you are using Kibana, then you probably have one replica for its index.

But there is nothing wrong to have a yellow cluster. What problem you want to solve?

6

I am not using Kibana. ElasticSearch is installed on the server and launched as a service.
ES is used to store some datas and displays them on a Google Maps API.
So i have a script to explore my database and fill ElasticSearch.
But when I launch this script, it works for 10min (maybe) and then i get this error message Elasticsearch\Common\Exceptions\ServerErrorResponseException
So i check the _cluster/health it appears to be yellow. Then i check my index localhost:9200/coloc and SURPRISE, the number_of_replicas is set to 1. I noted that ES works perfectly if my index's replicas stay to 0. But it doesnt... I am very lost :confused:

7

Why do you mean by "work perfectly"?
What does not work when the cluster is yellow?

8

Yesterday, during all the process of my script, the cluster stayed to green state. The script has finished to load my datas in ES without any problems and my markers were set on my Google maps api.

Today, nothing works... If i launch the script i get the error "Elasticsearch\Common\Exceptions\ServerErrorResponseException" and the cluster state becomes yellow.

You talked about template earlier, how this can help me ?

Add : ES receives a part of my data then ServerErrorResponse... I think ES is not stable but don't know why
[root@stock /]# curl localhost:9200/_cluster/health?pretty { "cluster_name" : "elasticsearch", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 5, "active_shards" : 5, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 5, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 50.0 }

[root@stock /]# curl localhost:9200/coloc

{"coloc":{"aliases":{},"mappings":{"annonces_propose":{"properties":{"location":{"type":"geo_point"},"price":{"type":"float"},"publish_date":{"type":"date","format":"yyyy-MM-dd HH:mm:ss"}}}},"settings":{"index":{"creation_date":"1524914951379","number_of_shards":"5","number_of_replicas":"1","uuid":"Z9UUTzcLRlq_hsrRTw7bBQ","version":{"created":"5000299"},"provided_name":"coloc"}}}}

9

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

1 Like
10

Everything seems ok here.

May be something wrong with your code.

Here is the link for index templates: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html

But this is not going to change anything IMO.

May be share also elasticsearch logs?

11

Except the filling data log i get that :

org.elasticsearch.env.ShardLockObtainFailedException: [coloc][4]: obtaining shard lock timed out after 0ms
        at org.elasticsearch.env.NodeEnvironment$InternalShardLock.acquire(NodeEnvironment.java:711) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.env.NodeEnvironment.shardLock(NodeEnvironment.java:630) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.env.NodeEnvironment.lockAllForIndex(NodeEnvironment.java:576) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.env.NodeEnvironment.deleteIndexDirectorySafe(NodeEnvironment.java:527) ~[elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.indices.IndicesService.deleteIndexStoreIfDeletionAllowed(IndicesService.java:642) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.indices.IndicesService.deleteIndexStore(IndicesService.java:629) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.indices.IndicesService.removeIndex(IndicesService.java:517) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.indices.IndicesService.deleteIndex(IndicesService.java:572) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.deleteIndices(IndicesClusterStateService.java:258) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:192) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.cluster.service.ClusterService.runTasksForExecutor(ClusterService.java:729) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.cluster.service.ClusterService$UpdateTask.run(ClusterService.java:915) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:238) [elasticsearch-5.0.2.jar:5.0.2]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:201) [elasticsearch-5.0.2.jar:5.0.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

and this from my last test script launching :

[2018-04-28T16:00:01,738][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]
[2018-04-28T16:02:32,513][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]
[2018-04-28T16:04:31,332][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]
[2018-04-28T16:06:31,559][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]
[2018-04-28T16:20:04,975][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]
[2018-04-28T16:22:23,157][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]
[2018-04-28T16:23:16,678][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[0], mappings [annonces_propose]
[2018-04-28T16:23:17,506][INFO ][o.e.c.r.a.AllocationService] [M32YSFl] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[coloc][4]] ...]).
[2018-04-28T16:23:17,823][INFO ][o.e.c.m.MetaDataMappingService] [M32YSFl] [coloc/r2_vcVFxSe-BpzyL3_XhdA] update_mapping [annonces_propose]
[2018-04-28T16:23:20,844][INFO ][o.e.c.m.MetaDataMappingService] [M32YSFl] [coloc/r2_vcVFxSe-BpzyL3_XhdA] update_mapping [annonces_propose]
[2018-04-28T16:32:17,376][INFO ][o.e.c.m.MetaDataCreateIndexService] [M32YSFl] [coloc] creating index, cause [api], templates [], shards [5]/[1], mappings [annonces_propose]

I don't really see an error that can cause the stop of my script. What do you think about it ? What kind of error i must search ? The file log is much too long to put it there.

12

You can share the full log on gist.github.com

And share exactly what your script is doing.

Also, please use instead latest 6.2 version if possible.

13

Hello,

Here is the link to my elasticsearch logs and the script I use to fill it.

14

Any idea why you have so many indices creation call?

Are you removing manually, then testing again?

Sadly I don't really read PHP code. But may be this is because you put in __construct() method the creation itself, which means to me that anytime you create an instance, it creates one index?
BTW number of shards could be set to 1 as well but I don't understand why this is not happening.
It's like you are calling:

PUT coloc

instead of

PUT coloc
{
  "settings": {
    // ...
  },
  "mappings": {
    // ...
  }
}
15

I was using an older version of ES before this one. Is it possible it use an old config file ?? I cleaned up all the server's folder before installing new version.

16

I don't know.

17

J'ai du nouveau! Le script ne plante plus. Toutes mes données sont récupérées mais (nouveauté) ne sont pas affichées à la demande... Je remarque que sous le dossier
/var/lib/elasticsearch/nodes/0/indices
de mon serveur, le dossier généré automatiquement change constamment de nom pendant que je remplis ES avec mes données... Est-ce un comportement normal ?

18

J'ai déplacé ta question dans Discussions en français.

le dossier généré automatiquement change constamment de nom pendant que je remplis ES avec mes données..

Peut-être mais de toute façon tu ne devrais pas aller farfouiller dans ce répertoire.

Utilise les API à la place. Genre:

GET _cat/indices?v
19

J'ai déplacé ta question dans Discussions en français.

Super merci, je ne me suis pas rendu compte de la langue dans laquelle j'écrivais.
Voilà la sortie de la commande

curl localhost:9200/_cat/coloc?v

[root@stock elasticsearch]# curl localhost:9200/_cat/indices?v
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   coloc chMicsd6QYmWNHsNKEdyaA   5   1          0            0       650b           650b

et 30 secondes plus tard....

[root@stock elasticsearch]# curl localhost:9200/_cat/indices?v
health status index uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   coloc K2dcfFjcQOu5OrDs-65KIw   5   1          0            0       650b           650b
20

Ca veut dire que tu détruis l'index à chaque run et que tu le reconstruis.