ElasticSearch 3 node fail over not working

Elastic Stack Elasticsearch
1

I have 3 node and config setup as shown below in all 3 es server. If anyone has similar fail over issue, please let me how did you resolved. I am implementing first time in Oracle PeopleSoft application.

elasticsearch.yml

server1
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true

server2
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true

server3
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true

2

Moved to #elasticsearch

What does not work?

3

The fail over not working. When I shut down server1 (Node1) the ES does not switch automatically to either of the 2 additional node in cluster.

Thanks
Qamar

4

Which client does not switch?

5

We have 2 server lets d01, d02 and d03. When I shutdown d01 search client not fail over to d2.

6

What kind of client it is?

7

Peoplesoft ERP.

8

I don't know what kind of Elasticsearch Client they are using.
Do you?

9

We are using ElasticSearch es2.3.2

10

That does not answer my question.
What is the elasticsearch client or library which is used by Peoplesoft to connect to elasticsearch cluster?

Anyway, here is what you can do:

  • Ask Peoplesoft support
  • Add a Load Balancer (LB) in front of elasticsearch nodes but if the LB fails you will have the same issue
  • Add a coordinating only node (master and data: false) and connect Peoplosoft to this node. But same as previous, if it fails, you will have the same issue.

Most of elasticsearch clients are supporting using multiple nodes, like the Java REST Client, Logstash or Beats.

11

If you have 3 master eligible nodes in the cluster, you need to set discovery.zen.minimum_master_nodes to 2 in order to avoid split-brain scenarios and data loss. Please see the documentation for further details.

12

Client info - PeopleSoft Integration Broker client. Please see url link below.

https://portal.pttmor.org/PSIGW/RESTListeningConnector/LUDNOD_PA

13

Cluster status shows this.

http://portal.pttmor.org:9200/_cluster/state?pretty

{
"cluster_name" : "ESCLUSTER",
"version" : 18,
"state_uuid" : "CYJlsergQdyeV5cjf4N3zQ",
"master_node" : "bvloXYVFQySNaA58pV8q-g",
"blocks" : { },
"nodes" : {
"bvloXYVFQySNaA58pV8q-g" : {
"name" : "node-1",
"transport_address" : "portal.pttmor.org/10.119.56.102:9300",
"attributes" : {
"rack" : "r1"
}
}
},
"metadata" : {
"cluster_uuid" : "q5ArCX4PR7K_VCAa68Qiow",
"templates" : { },
"indices" : {
"ptportalregistry_hcmtst" : {
"state" : "open",
"settings" : {
"index" : {
"creation_date" : "1515271526955",
"analysis" : {
"filter" : {
"orcl_es_greek_lowercase" : {
"type" : "lowercase",
"language" : "greek"
},
"orcl_es_turkish_lowercase" : {
"type" : "lowercase",
"language" : "turkish"
},
"orcl_es_english_possessive_stemmer" : {
"type" : "stemmer",
"language" : "possessive_english"
},
"orcl_es_phonetic" : {
"replace" : "true",
"type" : "phonetic",
"encoder" : "double_metaphone"
},
"orcl_es_cjk_bigrams_filter" : {
"output_unigrams" : "true",
"type" : "cjk_bigram"
},
"orcl_es_italian_elision" : {
"type" : "elision",
"articles" : [ "c", "l", "all", "dall", "dell", "nell", "sull", "coll", "pell", "gl", "agl", "dagl", "degl", "negl", "sugl", "un", "m", "t", "s", "v", "d" ]
},
"orcl_es_stemmer_eng" : {
"type" : "stemmer",
"language" : "english"
},
"orcl_es_french_elision" : {
"type" : "elision",
"articles" : [ "l", "m", "t", "qu", "n", "s", "j", "d", "c", "jusqu", "quoiqu", "lorsqu", "puisqu" ],
"articles_case" : "true"
},
"orcl_es_synonym" : {
"type" : "synonym",
"synonyms_path" : "analysis/synonym_grammer.txt"
}
},
"analyzer" : {
"orcl_es_custom_eng_stop" : {
"filter" : [ "standard", "lowercase", "orcl_es_synonym", "orcl_es_stemmer_eng" ],
"type" : "custom",
"stopwords" : "english",
"tokenizer" : "standard"
},
"orcl_es_custom_keyword" : {
"filter" : [ "lowercase" ],
"type" : "custom",
"tokenizer" : "keyword"
},
"orcl_es_custom_lowercase" : {
"filter" : [ "lowercase" ],
"type" : "custom",
"tokenizer" : "standard"
},
"orcl_es_path_hierarchy" : {
"type" : "custom",
"tokenizer" : "path-tokenizer"
},
"orcl_es_phonetic_analyzer" : {
"filter" : "orcl_es_phonetic",
"tokenizer" : "standard"
},
"orcl_es_custom_eng" : {
"filter" : [ "orcl_es_english_possessive_stemmer", "standard", "lowercase", "orcl_es_synonym", "orcl_es_stemmer_eng" ],
"type" : "custom",
"stopwords" : "none",
"tokenizer" : "standard"
}
},

14

I have setup as as you mention earlier.

discovery.zen.ping.unicast.hosts: ["server1.pttmor.org", "server2.pttmor.org", "server3.pttmor.org"]
discovery.zen.minimum_master_nodes: 2

My question is do I need to keep same in all 3 server?

Thanks for you all help.

15

You did that in the server. Not on the client side.

16

I can't access to your private network I think. But I guess you are using that: https://blogs.oracle.com/peopletools/elasticsearch-for-peoplesoft-now-available

May be it's better to ask Oracle about how to configure this tool?

18

This is my current setup, but when i shut down server 1 search did not work. Seems like cluster not working.

Server1

node.master: true

node.data: true

Server2

node.master: true

node.data: true

Server3

node.master: true

node.data: true

19

Please format your code using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

Please edit your post and share the logs of the 3 nodes.
What gives GET _cat/nodes?v on one of the nodes which is still running?

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.