# Elasticsearch not starting on Second Run... Voting Configuration and Quorum Issue **URL:** \ **Category:** Elasticsearch\ **Created:** [July 8, 2019, 7:14pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416 "2019-07-08T19:14:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![oriondark](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oriondark](https://discuss.elastic.co/u/oriondark)\ **Post date:** [July 8, 2019, 7:14pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/1 "2019-07-08T19:14:44Z")
Startup error: (not on initial run of new elasticsearch instance) "org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot start with [discovery.type] set to [single-node] when local node {878bc9e4fbfa}{fwjpwdIvTAet6GrkBXymjg}{dTcQaACOQcO7GmOJds2Gkw}{172.18.0.2}{172.18.0.2:9300}{ml.machine\_memory=8341348352, xpack.installed=true, ml.max\_open\_jobs=20} does not have quorum in voting configuration VotingConfiguration{AICGKQZ1TZux3KwYt0YhfA}.. ---.yml rollup (elk stack flavor from GitHub ) elasticsearch: build: context: elasticsearch/ args: ELK\_VERSION: $ELK\_VERSION volumes: - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro - ./data:/usr/share/elasticsearch/data ports: - "9200:9200" - "9300:9300" environment: ES\_JAVA\_OPTS: "-Xmx6g -Xms2g" ELASTIC\_PASSWORD: changeme ulimits: memlock: soft: -1 hard: -1 networks: - elk ---elasticsearch.yml cluster.name: "docker-cluster" network.host: 0.0.0.0 node.ml: false discovery.type: single-node xpack.license.self\_generated.type: basic So, first instance always runs. Once I get data pushed into it, and attempt to restart, it fails. I've tried a variety of things with the config, but to no avail. I'm assuming its something to do with the bootstrap checks and/or the existence of ml stuff (possibly from Xpack)? Any ideas on this? My main goal is to get a dev instance going that I can load up, destroy, test, and generally push around. Thanks so much! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 8, 2019, 9:29pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/2 "2019-07-08T21:29:56Z")
The issue is that this node has unique ID `fwjpwdIvTAet6GrkBXymjg` but previously it belonged to a cluster with a node with ID `AICGKQZ1TZux3KwYt0YhfA` and this latter node is known to have the freshest cluster state. Either there were multiple nodes in the cluster at one time, or else the contents of the data path are not persisting across restarts. But that'd be pretty weird because it's not that _none_ of the contents of the data path is persisting across restarts either: some of the data is surviving, but not all of it. It's vitally important that the whole data path persists across restarts. It looks like you're using Docker, but I don't recognise the Dockerfile. Where have you got that from? Also please don't use the password `changeme`. ---
**Author:** ![oriondark](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oriondark](https://discuss.elastic.co/u/oriondark)\ **Post date:** [July 9, 2019, 2:17pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/3 "2019-07-09T14:17:03Z")
David, Thanks so much for the reply. I got the elk stack images for docker from [https://github.com/deviantony/docker-elk](https://github.com/deviantony/docker-elk). There should never have been more than 1 node from my understanding, and the data does persist between restarts. To complicate things, I am running on Docker Desktop (for Windows) and the images and data are on a USB drive. Smile. I was able to push everything through on the old 6.x versions with the same GitHub pieces, but keep running into the Voting Quorum thing. I've tried to dive into the documentation, but don't exactly understand how to either bypass the bootstrap checks (doesn't sound like that's the issue), or, as you are helping me discover, harden (or fix) the node IDs between restarts. Any further suggestions/help would be great! Thanks so much for the reply. Love the product. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 9, 2019, 2:26pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/4 "2019-07-09T14:26:29Z")
Certainly a nonstandard configuration, and a nonstandard Docker config too. Which of the node IDs is the "right" one? I.e. when you start the node the first time it'll log a line like this: ```auto [2019-07-09T15:14:02,382][INFO][o.e.n.Node] [node-0] node name [node-0], node ID [WvFG5LtFSLiqfmM1IeOPHg], cluster name [elasticsearch] ``` Here `WvFG5LtFSLiqfmM1IeOPHg` is my node ID. When you start it up the second time, does the original node ID match the new node ID or does it match the one in the `VotingConfiguration` (or, even more strangely, does it match neither?) When you say "the data does persist between restarts" can you be more precise, and share a list of the full paths of the files that are put on persistent storage? When you restart, do you see the last-modified dates of these files change? ---
**Author:** ![oriondark](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oriondark](https://discuss.elastic.co/u/oriondark)\ **Post date:** [July 9, 2019, 2:50pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/5 "2019-07-09T14:50:59Z")
Sorry...new ids, be trying various things. Here's the log line: elasticsearch\_1 | {"type": "server", "timestamp": "2019-07-09T14:27:56,713+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "2580ec46aa93", "message": "node name [2580ec46aa93], node ID [PV9cpNj7Sxax7lyO3\_mzmw], cluster name [docker-cluster]" } And the accompanying Voting Quorum piece: elasticsearch\_1 | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot start with [discovery.type] set to [single-node] when local node {2580ec46aa93}{PV9cpNj7Sxax7lyO3\_mzmw}{u9dwInaTTK-qdlm2naqXlA}{172.18.0.2}{172.18.0.2:9300}{ml.machine\_memory=6227419136, xpack.installed=true, ml.max\_open\_jobs=20} does not have quorum in voting configuration VotingConfiguration{6yNfx-jYQRm649gF67sa9g}", Both the u9dw.. identifier and the 6yNf.. identifier do not occur anywhere else in the output. Data path: /data/nodes/0/[\_state][node.lock][indices]/[all indices with uuid-type names/state/0, etc... everything lives in one place. Appears like 'state' pieces [nodes\0\indices\tW163-2tTDmFfzgOOcJxyQ\_state\state-xx.st]have been modified since last good run (when data was importing) data\nodes\0\indices\KM4GWJjKTfq\_6MX9FkmUzw\0\index - contains 20gb of cfe, si, cfs, etc files - modified at last good run. Did I get everything? Thanks again. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 9, 2019, 3:38pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/6 "2019-07-09T15:38:57Z")
Sorry, I will clarify. As I understand it, the first time you start this node up (empty data directory) it works, but then the second time it doesn't. I would like to see the log lines containing the node IDs from **both** runs, as well as the `cannot start with [discovery.type] set to [single-node]` line. The two files to look at in particular are `data/nodes/0/_state/global-NNN.st` and `data/nodes/0/_state/node-NNN.st` (for decimal numbers `NNN`). They should be replaced with higher-numbered files each startup. Are they? ---
**Author:** ![oriondark](https://avatars.discourse-cdn.com/v4/letter/o/a9adbd/32.png) [@oriondark](https://discuss.elastic.co/u/oriondark)\ **Post date:** [July 10, 2019, 2:53pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/7 "2019-07-10T14:53:01Z")
Sorry for the delay. Had to travel yesterday. Anywho, here are some of the logs: - first three starts (create, kill --restart, index-- restart data) worked... after data, things failed. First Run elasticsearch\_1 | {"type": "server", "timestamp": "2019-07-10T13:02:22,695+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "16278c81ca88", "message": "node name [16278c81ca88], node ID [8ZqcR-3SS8aldEtzbbGbaQ], cluster name [docker-cluster]" } Second Run - No Indices, No data elasticsearch\_1 | {"type": "server", "timestamp": "2019-07-10T13:06:41,243+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "16278c81ca88", "message": "node name [16278c81ca88], node ID [8ZqcR-3SS8aldEtzbbGbaQ], cluster name [docker-cluster]" } Third run - Indices elasticsearch\_1 | {"type": "server", "timestamp": "2019-07-10T13:12:15,907+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "16278c81ca88", "message": "node name [16278c81ca88], node ID [8ZqcR-3SS8aldEtzbbGbaQ], cluster name [docker-cluster]" } Fourth run -indices, data, and error elasticsearch\_1 | {"type": "server", "timestamp": "2019-07-10T14:40:36,614+0000", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "16278c81ca88", "message": "node name [16278c81ca88], node ID [\_WYf379jRwyIZn95s0hZBw], cluster name [docker-cluster]" } elasticsearch\_1 | "stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: cannot start with [discovery.type] set to [single-node] when local node {16278c81ca88}{\_WYf379jRwyIZn95s0hZBw}{5qYSXsNXSaSOHp4amJJJqA}{172.18.0.2}{172.18.0.2:9300}{ml.machine\_memory=6227410944, xpack.installed=true, ml.max\_open\_jobs=20} does not have quorum in voting configuration VotingConfiguration{8ZqcR-3SS8aldEtzbbGbaQ}", global and node files appear to be going up every time (decimal numbers). I have the full logs of each run. I can do another run and save the \_indices piece of each run. It appears that at some point, after data, the clusterId changes. Thanks again for all the help. Please let me know if I can get you any more info. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 10, 2019, 3:38pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/8 "2019-07-10T15:38:15Z")
Huh. I wasn't expecting to see that. ```auto node ID [8ZqcR-3SS8aldEtzbbGbaQ] ... node ID [_WYf379jRwyIZn95s0hZBw] ``` The node ID is changing. This is stored in `data/nodes/0/_state/node-NNN.st`, and we only ever create a new node ID if the file isn't there. It's a binary file, but it's tiny and contains the node ID in plain text so you should be able to see it in a text editor. The only explanation I can think of is that something is removing this file between runs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 7, 2019, 3:38pm UTC](https://discuss.elastic.co/t/elasticsearch-not-starting-on-second-run-voting-configuration-and-quorum-issue/189416/9 "2019-08-07T15:38:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.