Installing Elasticsearch on Azure Vm Scaleset


(vijayakrishna Gade) #1

i am trying to Configure Elasticsearch on Red Hat Enterprise Linux Server release 7.3 (Maipo) with Azure VM Scaleset, service is unable to start, can someone please help me with it, Thanks.

sudo systemctl status elasticsearch.service
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-10-22 20:45:04 UTC; 6s ago
Docs: http://www.elastic.co
Process: 18600 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 18600 (code=exited, status=1/FAILURE)

Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:127)
Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: at org.elasticsearch.cli.Command.main(Command.java:90)
Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:86)
Oct 22 20:45:04 s00199cdxnginxtestna2000000 elasticsearch[18600]: Refer to the log for complete error details.
Oct 22 20:45:04 s00199cdxnginxtestna2000000 systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Oct 22 20:45:04 s00199cdxnginxtestna2000000 systemd[1]: Unit elasticsearch.service entered failed state.
Oct 22 20:45:04 s00199cdxnginxtestna2000000 systemd[1]: elasticsearch.service failed.

cat elasticsearch.yml

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

#network.host: 192.168.0.1

Set a custom port for HTTP:

#http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when new node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.zen.ping.unicast.hosts: ["host1", "host2"]

Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):

#discovery.zen.minimum_master_nodes:

For more information, consult the zen discovery module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true


(Russ Cam) #2

How are you deploying Elasticsearch and configuring it to run on a VM Scale set? Looking at the elasticsearch.yml posted for example,

  1. the cluster name for each VM in the Scale set needs to be the same
  2. Each node needs to know how to communicate with other nodes to form a cluster, using one or more hosts in discovery.zen.ping.unicast.hosts.

Can you share more details on how you are configuring and deploying?


(vijayakrishna Gade) #3

Initially am trying to configure on first node/one Node in the cluster.

By following below documentation:


(Russ Cam) #4

That guide is very general, and may not be sufficient without further changes, to deploy to Azure in a VM scale set.

You may want to take a look at the ARM template repository that can deploy the Elastic Stack to Azure, using Availability Sets but not VM Scale sets. You may then want to fork the repository to add VM scale sets, using the Azure quickstart template as a guide.


(vijayakrishna Gade) #5

Thanks for Documentation.

i am following ARM template repository to configure elastic stack.

i have existing Virtual network and subnet, doesn't have permission to create virtual networks in subscription.
Existing virtual network and subnet details mentioned in the password.parameters.json file, getting below issue.

az group deployment create --resource-group elasticsearch --template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json --parameters @password.parameters.json
Deployment failed. Correlation ID: e7ef6e5e-02e3-4513-98bf-a0ac89f19944. {
"error": {
"code": "InvalidTemplateDeployment",
"message": "The template deployment failed with error: 'Authorization failed for template resource 'es-app-gateway-ip' of type 'Microsoft.Network/publicIPAddresses'. The client 'azureaccount@gmail.com' with object id '95f743cb-5237-4ebf-7155-017cf8135ed2' does not have permission to perform action 'Microsoft.Network/publicIPAddresses/write' at scope '/subscriptions/75aa954b-5b6c-1b2a-a5ef-12df3hyt678/resourceGroups/elasticsearch/providers/Microsoft.Network/publicIPAddresses/es-app-gateway-ip'.'."

here is the reference of password.parameters.json

cat password.parameters.json
{
"artifactsBaseUrl":{"value":"https://raw.githubusercontent.com/elastic/azure-marketplace/master/src"},
"esVersion":{"value":"6.2.4"},
"esClusterName":{"value":"es-cluster"},
"loadBalancerType":{"value":"gateway"},
"azureCloudPlugin":{"value":"No"},
"azureCloudStorageAccountName":{"value":""},
"azureCloudStorageAccountResourceGroup":{"value":""},
"xpackPlugins":{"value":"Yes"},
"esAdditionalPlugins": { "value":""},
"esAdditionalYaml": { "value":""},
"esHeapSize": { "value":0},
"esHttpCertBlob": { "value":""},
"esHttpCertPassword": { "value":""},
"esHttpCaCertBlob": { "value":""},
"esHttpCaCertPassword": { "value":""},
"esTransportCaCertBlob": { "value":""},
"esTransportCaCertPassword": { "value":""},
"esTransportCertPassword": { "value":""},
"samlMetadataUri": { "value":""},
"samlServiceProviderUri": { "value":""},
"kibana":{"value":"Yes"},
"vmSizeKibana":{"value":"Standard_D1"},
"vmKibanaAcceleratedNetworking":{"value":"Default"},
"kibanaCertBlob":{"value":""},
"kibanaKeyBlob":{"value":""},
"kibanaKeyPassphrase":{"value":""},
"kibanaAdditionalYaml": { "value":""},
"logstash": { "value":"No"},
"vmSizeLogstash": { "value":"Standard_D1"},
"vmLogstashAcceleratedNetworking":{"value":"Default"},
"logstashHeapSize": { "value": 0 },
"logstashConf": { "value": "" },
"logstashKeystorePassword": { "value": "" },
"logstashAdditionalPlugins": { "value": "" },
"logstashAdditionalYaml": { "value": "" },
"jumpbox":{"value":"No"},
"vmHostNamePrefix":{"value":""},
"vmSizeDataNodes":{"value":"Standard_D1"},
"vmDataNodeAcceleratedNetworking":{"value":"Default"},
"vmDataDiskCount":{"value":40},
"vmDataDiskSize":{"value":"Small"},
"vmDataNodeCount":{"value":3},
"storageAccountType":{"value":"Default"},
"dataNodesAreMasterEligible":{"value":"Yes"},
"vmSizeMasterNodes":{"value":"Standard_DS2"},
"vmMasterNodeAcceleratedNetworking":{"value":"Default"},
"vmClientNodeCount":{"value":0},
"vmSizeClientNodes":{"value":"Standard_D1"},
"vmClientNodeAcceleratedNetworking":{"value":"Default"},
"adminUsername":{"value":"elasticsearch"},
"authenticationType":{"value":"password"},
"sshPublicKey":{"value":""},
"adminPassword":{"value":"xxxxxxxx"},
"securityBootstrapPassword":{"value":"xxxxxxxx"},
"securityAdminPassword":{"value":"xxxxxxxx"},
"securityReadPassword":{"value":"xxxxxxxx"},
"securityKibanaPassword":{"value":"xxxxxxxx"},
"securityLogstashPassword":{"value":"xxxxxxxx"},
"securityBeatsPassword":{"value":"xxxxxxxx"},
"vNetNewOrExisting": {"value":"existing"},
"vNetName": {"value": "myvent1"},
"vNetExistingResourceGroup": {"value": "myvnetrg"},
"vNetNewAddressPrefix": {"value": ""},
"vNetLoadBalancerIp": {"value": "10.18.xx.xx"},
"vNetClusterSubnetName": {"value": "mysubnet"},
"vNetNewClusterSubnetAddressPrefix": {"value": ""},
"vNetAppGatewaySubnetName": {"value": "mysubnet"},
"vNetNewAppGatewaySubnetAddressPrefix": {"value": "10.18.xx.xx/25"},
"appGatewayTier": {"value":"Standard"},
"appGatewaySku": {"value":"Small"},
"appGatewayCount": {"value":1},
"appGatewayCertBlob": {"value":""},
"appGatewayCertPassword": {"value":""},
"appGatewayWafStatus": {"value":"Disabled"},
"appGatewayWafMode": {"value":"Detection"},
"appGatewayEsHttpCertBlob": { "value": ""}
}


(Russ Cam) #6

The account you're using doesn't have permission to create a public IP address for Application Gateway.

Additionally, Application Gateway needs to deploy to its own subnet in the vnet, and that subnet can only contain other Application Gateway..

Give it a try with an account that has sufficient permissions to create all resources. If you're targeting an existing vnet and wish to use Application Gateway, also be sure that the subnet for Application Gateway exists before starting the deployment.


(vijayakrishna Gade) #7

Hi Russ,

I have installed Azure Scaleset with Azure internal Loadbalancer and it has 2 VM's in the cluster.

Later i have installed Elsticsearch-6.x, kibana-6.x on both the machines separately and enabled SSL with x-pack on both the machines.

LB IP: 10.0.0.1
VM1: 10.0.0.2
VM2: 10.0.0.3

Both the machines has Elsticsearch on port 9200 and Kibana on 5601 ports running.

i have added LB rule to forward traffic from port 443 to 9200, as well as Healthprobe, and also added port 9200 and 5601 on machine in the iptables to accept.

Right now when use curl command from outside of cluster VM1 ip: 10.0.0.2 (curl -I 'https://10.0.0.2:9200') is able to access.

but when i use LB ip elastic service is not able to reach through LB ip. (curl -I 'https://10.0.0.1:443') or (curl -I 'https://10.0.0.1:9200')

Same thing with Kibana service as well with port 5601.

i have added NSG rules ANY-ANY for testing Purpose.

please suggest me how do i enable access to elasticsearch and Kibana through LB ip, Thanks.

elasticsearch.yml

node.name: node1
network.host: node1.elastic.test.com
xpack.ssl.key: certs/node1.key
xpack.ssl.certificate: certs/node1.crt
xpack.ssl.certificate_authorities: certs/ca.crt
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
discovery.zen.ping.unicast.hosts: [ 'node1.elastic.test.com', 'node2.elastic.test.com']
node.max_local_storage_nodes: 2
xpack.security.enabled: true


(vijayakrishna Gade) #8

I have enabled Healthprobe Kibana and Elsticsearch is able to access using LB IP, Thanks.


(vijayakrishna Gade) #9

I have an another issue with connecting Kibana to elasticsearch, trying to generate password getting below issue:

bin/elasticsearch-setup-passwords auto -u "https://node1.elastic.test.com:9200"

Unexpected response code [403] from calling GET https://node1.elastic.test.com:9200/_xpack/security/_authenticate?pretty
It doesn't look like the X-Pack security feature is available on this Elasticsearch node.
Please check if you have installed a license that allows access to X-Pack Security feature.

ERROR: X-Pack Security is not available.

Kibana log:

"type":"log","@timestamp":"2018-11-01T18:56:58Z","tags":["warning","elasticsearch","admin"],"pid":72147,"message":"No living connections"}
{"type":"error","@timestamp":"2018-11-01T18:56:59Z","tags":["connection","client","error"],"pid":72147,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket. (_tls_wrap.js:876:25)\n at emitOne (events.js:121:20)\n at TLSSocket.emit (events.js:211:7)\n at _handle.close (net.js:557:12)\n at Socket.done (_tls_wrap.js:356:7)\n at Object.onceWrapper (events.js:315:30)\n at emitOne (events.js:121:20)\n at Socket.emit (events.js:211:7)\n at TCP._handle.close [as _onclose] (net.js:557:12)","code":"ECONNRESET"},"message":"socket hang up"}
{"type":"log","@timestamp":"2018-11-01T18:57:01Z","tags":["warning","elasticsearch","admin"],"pid":72147,"message":"Unable to revive connection: https://node1.elastic.test.com:9200/"}
{"type":"log","@timestamp":"2018-11-01T18:57:01Z","tags":["warning","elasticsearch","admin"],"pid":72147,"message":"No living connections"}
{"type":"log","@timestamp":"2018-11-01T18:57:03Z","tags":["warning","elasticsearch","admin"],"pid":72147,"message":"Unable to revive connection: https://node1.elastic.test.com:9200/"}


(vijayakrishna Gade) #10

Password Authentication issue has been resolved.

i see below error in kibana.stdout file, please help me.

{"type":"error","@timestamp":"2018-11-02T01:17:52Z","tags":["connection","client","error"],"pid":26703,"level":"error","error":{"message":"socket hang up","name":"Error","stack":"Error: socket hang up\n at TLSSocket. (_tls_wrap.js:847:25)\n at emitOne (events.js:101:20)\n at TLSSocket.emit (events.js:188:7)\n at _handle.close (net.js:497:12)\n at Socket.done (_tls_wrap.js:332:7)\n at Socket.g (events.js:292:16)\n at emitOne (events.js:101:20)\n at Socket.emit (events.js:188:7)\n at TCP._handle.close [as _onclose] (net.js:497:12)","code":"ECONNRESET"},"message":"socket hang up"}


(Russ Cam) #11

Ensure

  1. Kibana is configured to communicate with Elasticsearch using TLS (as it looks like TLS has been configured on the Elasticsearch HTTP layer)
  2. if X-Pack Security is enabled (which it looks like it is), that kibana will use a username/password when it communicates with Elasticsearch; the built-in kibana user can be used here.

(vijayakrishna Gade) #12

Hi Russ,

Apart from Socket issue, we have configured fluentd on kubernetes cluster to forward all the Kubernetes logs and Pod logs to Elasticsearch, it was working before enabling SSL on Elasticsearch, once SSL enabled then it's failing to not able to connect to Elastic server:

Fluentd Log:

2018-11-02 03:18:41 +0000 [warn]: #0 failed to flush the buffer. retry_time=12 next_retry_seconds=2018-11-02 03:18:41 +0000 chunk="579a50e3a58799f23887edc6f24e6ea5" error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>"10.0.0.1", :port=>9200, :scheme=>"http"})!"
2018-11-02 03:18:41 +0000 [warn]: #0 suppressed same stacktrace

Here is the Fluentd.conf looks like:

<match kubernetes.**>
@type copy

@type elasticsearch
server 10.0.0.1
port 9200
protocol https
verify false
index log-nprod
sourcetype _json
source kubernetes
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s


(vijayakrishna Gade) #13

2018-11-02 03:54:59 +0000 [warn]: #0 failed to flush the buffer. retry_time=5 next_retry_seconds=2018-11-02 03:54:59 +0000 chunk="579a67dbf204491a741c26a6e46d4dc4" error_class=Faraday::SSLError error="SSL_connect returned=1 errno=0 state=error: certificate verify failed (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:\n\n Excon.defaults[:ssl_ca_path] = path_to_certs\n ENV['SSL_CERT_DIR'] = path_to_certs\n Excon.defaults[:ssl_ca_file] = path_to_file\n ENV['SSL_CERT_FILE'] = path_to_file\n Excon.defaults[:ssl_verify_callback] = callback\n (see OpenSSL::SSL::SSLContext#verify_callback)\nor:\n Excon.defaults[:ssl_verify_peer] = false (less secure).\n"
2018-11-02 03:54:59 +0000 [warn]: #0 suppressed same stacktrace

fluentd.conf

<match kubernetes.**>
@type copy

@type elasticsearch
host 10.0.0.1
port 80
scheme https
ca_file /cert_blog/certs/ca/ca.crt
client_cert /cert_blog/certs/node1/node1.cert
client_key /cert_blog/certs/node1/node1.key
client_key_pass xxxxxx
verify false
index log-nprod
sourcetype _json
source kubernetes
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s


(vijayakrishna Gade) #14

Hi i am getting below error while authenticating leasticsearch from fluentd, please suggest me, thanks

2018-11-05 20:33:21 +0000 [warn]: #0 failed to flush the buffer. retry_time=7 next_retry_seconds=2018-11-05 20:33:20 +0000 chunk="579f0c3f1d7c6bf67d5a974d4efc94cf" error_class=Fluent::Plugin::ElasticsearchOutput::ConnectionFailure error="Can not reach Elasticsearch cluster ({:host=>"10.0.0.1", :port=>80, :scheme=>"https", :user=>"elastic", :password=>"obfuscated"})!"
2018-11-05 20:33:21 +0000 [warn]: #0 suppressed same stacktrace


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.