I am new to Elasticsearch and am setting up a new environment. I read through the posts and it seems my issue may be a bit different. When I deploy the new elasticsearch marketplace template through the UI, I am getting a lot of failures. Elasticsearch Web UI does appear to be up, but it will not accept the elastic password I set on deployment.
Here is the full Error Details:
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-debug for usage details.","details":[{"code":"Conflict","message":"{\r\n "status": "Failed",\r\n "error": {\r\n "code": "ResourceDeploymentFailure",\r\n "message": "The resource operation completed with terminal provisioning state 'Failed'.",\r\n "details": [\r\n {\r\n "code": "VMExtensionProvisioningError",\r\n "message": "VM has reported a failure when processing extension 'script'. Error message: \"Enable failed: failed to execute command: command terminated with exit status=10\n[stdout]\n\n[19032019-13:20:31] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 20/60\n[19032019-13:20:36] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 21/60\n[19032019-13:20:41] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 22/60\n[19032019-13:20:47] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 23/60\n[19032019-13:20:52] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 24/60\n[19032019-13:20:57] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 25/60\n[19032019-13:21:03] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 26/60\n[19032019-13:21:08] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 27/60\n[19032019-13:21:13] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 28/60\n[19032019-13:21:19] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 29/60\n[19032019-13:21:24] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 30/60\n[19032019-13:21:29] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 31/60\n[19032019-13:21:35] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 32/60\n[19032019-13:21:40] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 33/60\n[19032019-13:21:46] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 34/60\n[19032019-13:21:51] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 35/60\n[19032019-13:21:56] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 36/60\n[19032019-13:22:02] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 37/60\n[19032019-13:22:07] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 38/60\n[19032019-13:22:12] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 39/60\n[19032019-13:22:18] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 40/60\n[19032019-13:22:23] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 41/60\n[19032019-13:22:28] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 42/60\n[19032019-13:22:34] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 43/60\n[19032019-13:22:39] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 44/60\n[19032019-13:22:44] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 45/60\n[19032019-13:22:50] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 46/60\n[19032019-13:22:55] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 47/60\n[19032019-13:23:01] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 48/60\n[19032019-13:23:06] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 49/60\n[19032019-13:23:11] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 50/60\n[19032019-13:23:17] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 51/60\n[19032019-13:23:22] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 52/60\n[19032019-13:23:27] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 53/60\n[19032019-13:23:33] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 54/60\n[19032019-13:23:38] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 55/60\n[19032019-13:23:43] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 56/60\n[19032019-13:23:49] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 57/60\n[19032019-13:23:54] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 58/60\n[19032019-13:24:00] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 59/60\n[19032019-13:24:05] [wait_for_started] seeing if node is up after sleeping 5 seconds, retry 60/60\n[19032019-13:24:05] [wait_for_started] never saw elasticsearch go up locally\n\n[stderr]\n\rExtracting templates from packages: 31%\rExtracting templates from packages: 62%\rExtracting templates from packages: 93%\rExtracting templates from packages: 100%\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nsent invalidate(passwd) request, exiting\nsent invalidate(group) request, exiting\nSynchronizing state of elasticsearch.service with SysV init with /lib/systemd/systemd-sysv-install...\nExecuting /lib/systemd/systemd-sysv-install enable elasticsearch\nCreated symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.\nrun-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables save\nrun-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables save\n\"."\r\n }\r\n ]\r\n }\r\n}"}]}
Hey @brock.griffin
[wait_for_started] seeing if node is up after sleeping 5
This indicates there has been an issue forming the cluster, because the node was not seen to come up on the VM. To understand what the issue is in this case, you will need to SSH into the specified node(s) and look at the logs on the machine. Check out the troubleshooting section in the docs.
I'd start with the /var/log/arm-install.log and the logs in /var/log/elasticsearch/, and post here the contents (be sure to sanitize any potential sensitive data before posting).
Sorry this has taken so long to reply. I had to drop this and was able to get back to it today.
It appears that there were two forces at play here. The first of which was that our internal DNS the servers were set to use were not working properly. I was able to remediate this by adding all the nodes to the /etc/hosts file.
The second issue, once Elastic started to respond, was the passwords passed in the Azure template build did not pass through on the actual build. Running bin/elasticsearch-setup-passwords interactive and plugging in the appropriate passwords worked.
System is now up, available and chewing on fresh data.
Good to hear you resolved it @brock.griffin 
The template assumes Azure DNS to resolve VMs on the network by hostname, so if you've configured your own DNS, you can forward requests to Azure DNS to resolve by hostname, or change the elasticsearch.yml configuration for each deployed node.
I think the second issue is a knock on effect of the first; within the template, built-in users are configured via the REST API once the cluster is up and running, so if it fails to form, the built-in users won't end up being configured.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.