ElasticSearch 3 node fail over not working


(Qamar Hussain) #1

I have 3 node and config setup as shown below in all 3 es server. If anyone has similar fail over issue, please let me how did you resolved. I am implementing first time in Oracle PeopleSoft application.

elasticsearch.yml

server1
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true

server2
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true

server3
discovery.zen.minimum_master_nodes: 1
node.master: true
node.data: true


(David Pilato) #2

Moved to #elasticsearch

What does not work?


(Qamar Hussain) #3

The fail over not working. When I shut down server1 (Node1) the ES does not switch automatically to either of the 2 additional node in cluster.

Thanks
Qamar


(David Pilato) #4

Which client does not switch?


(Qamar Hussain) #5

We have 2 server lets d01, d02 and d03. When I shutdown d01 search client not fail over to d2.


(David Pilato) #6

What kind of client it is?


(Qamar Hussain) #7

Peoplesoft ERP.


(David Pilato) #8

I don't know what kind of Elasticsearch Client they are using.
Do you?


(Qamar Hussain) #9

We are using ElasticSearch es2.3.2


(David Pilato) #10

That does not answer my question.
What is the elasticsearch client or library which is used by Peoplesoft to connect to elasticsearch cluster?

Anyway, here is what you can do:

  • Ask Peoplesoft support
  • Add a Load Balancer (LB) in front of elasticsearch nodes but if the LB fails you will have the same issue
  • Add a coordinating only node (master and data: false) and connect Peoplosoft to this node. But same as previous, if it fails, you will have the same issue.

Most of elasticsearch clients are supporting using multiple nodes, like the Java REST Client, Logstash or Beats.


(Christian Dahlqvist) #11

If you have 3 master eligible nodes in the cluster, you need to set discovery.zen.minimum_master_nodes to 2 in order to avoid split-brain scenarios and data loss. Please see the documentation for further details.


(Qamar Hussain) #12

Client info - PeopleSoft Integration Broker client. Please see url link below.

https://portal.pttmor.org/PSIGW/RESTListeningConnector/LUDNOD_PA


(Qamar Hussain) #13

Cluster status shows this.

http://portal.pttmor.org:9200/_cluster/state?pretty

{
"cluster_name" : "ESCLUSTER",
"version" : 18,
"state_uuid" : "CYJlsergQdyeV5cjf4N3zQ",
"master_node" : "bvloXYVFQySNaA58pV8q-g",
"blocks" : { },
"nodes" : {
"bvloXYVFQySNaA58pV8q-g" : {
"name" : "node-1",
"transport_address" : "portal.pttmor.org/10.119.56.102:9300",
"attributes" : {
"rack" : "r1"
}
}
},
"metadata" : {
"cluster_uuid" : "q5ArCX4PR7K_VCAa68Qiow",
"templates" : { },
"indices" : {
"ptportalregistry_hcmtst" : {
"state" : "open",
"settings" : {
"index" : {
"creation_date" : "1515271526955",
"analysis" : {
"filter" : {
"orcl_es_greek_lowercase" : {
"type" : "lowercase",
"language" : "greek"
},
"orcl_es_turkish_lowercase" : {
"type" : "lowercase",
"language" : "turkish"
},
"orcl_es_english_possessive_stemmer" : {
"type" : "stemmer",
"language" : "possessive_english"
},
"orcl_es_phonetic" : {
"replace" : "true",
"type" : "phonetic",
"encoder" : "double_metaphone"
},
"orcl_es_cjk_bigrams_filter" : {
"output_unigrams" : "true",
"type" : "cjk_bigram"
},
"orcl_es_italian_elision" : {
"type" : "elision",
"articles" : [ "c", "l", "all", "dall", "dell", "nell", "sull", "coll", "pell", "gl", "agl", "dagl", "degl", "negl", "sugl", "un", "m", "t", "s", "v", "d" ]
},
"orcl_es_stemmer_eng" : {
"type" : "stemmer",
"language" : "english"
},
"orcl_es_french_elision" : {
"type" : "elision",
"articles" : [ "l", "m", "t", "qu", "n", "s", "j", "d", "c", "jusqu", "quoiqu", "lorsqu", "puisqu" ],
"articles_case" : "true"
},
"orcl_es_synonym" : {
"type" : "synonym",
"synonyms_path" : "analysis/synonym_grammer.txt"
}
},
"analyzer" : {
"orcl_es_custom_eng_stop" : {
"filter" : [ "standard", "lowercase", "orcl_es_synonym", "orcl_es_stemmer_eng" ],
"type" : "custom",
"stopwords" : "english",
"tokenizer" : "standard"
},
"orcl_es_custom_keyword" : {
"filter" : [ "lowercase" ],
"type" : "custom",
"tokenizer" : "keyword"
},
"orcl_es_custom_lowercase" : {
"filter" : [ "lowercase" ],
"type" : "custom",
"tokenizer" : "standard"
},
"orcl_es_path_hierarchy" : {
"type" : "custom",
"tokenizer" : "path-tokenizer"
},
"orcl_es_phonetic_analyzer" : {
"filter" : "orcl_es_phonetic",
"tokenizer" : "standard"
},
"orcl_es_custom_eng" : {
"filter" : [ "orcl_es_english_possessive_stemmer", "standard", "lowercase", "orcl_es_synonym", "orcl_es_stemmer_eng" ],
"type" : "custom",
"stopwords" : "none",
"tokenizer" : "standard"
}
},


(Qamar Hussain) #14

I have setup as as you mention earlier.

discovery.zen.ping.unicast.hosts: ["server1.pttmor.org", "server2.pttmor.org", "server3.pttmor.org"]
discovery.zen.minimum_master_nodes: 2

My question is do I need to keep same in all 3 server?

Thanks for you all help.


(David Pilato) #15

You did that in the server. Not on the client side.


(David Pilato) #16

I can't access to your private network I think. But I guess you are using that: https://blogs.oracle.com/peopletools/elasticsearch-for-peoplesoft-now-available

May be it's better to ask Oracle about how to configure this tool?


(Qamar Hussain) #18

This is my current setup, but when i shut down server 1 search did not work. Seems like cluster not working.

Server1

node.master: true

node.data: true

Server2

node.master: true

node.data: true

Server3

node.master: true

node.data: true


(David Pilato) #19

Please format your code using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

Please edit your post and share the logs of the 3 nodes.
What gives GET _cat/nodes?v on one of the nodes which is still running?


(system) #20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.