Weird error when putting template for index

Elastic Stack Elasticsearch
1

I have a hard time understanding the behavior so maybe someone here has seen it and can shed some light as to what is actually happening.

I'm using elasticsearch.js to push templates to indices. I use client.indices.putTemplate() function to do so. My ES cluster is behind nginx and I setup the ES client to connect to nginx using https on port 443. Nginx will then strip ssl and forward to a client node in the cluster.

All of the setup works fine for searching and basic usage of ES. Every time I try to associate a new template to an index, I get connection failure to https://:9200. Notice the wrong port. I don't quite understand why the ES client has to connect again to hosts that are behind the nginx gateway using either the wrong protocol (https instead of http) or the wrong port (9200 instead of 443).

If I change my configuration to connect the ES client to plain http (http://:9200) everything works fine. If I connect using https://:443, I get the following error:

PUT https://<ip>:9200/_template/template_mapping?order=2&create=true&master_timeout=600000 => socket hang up at Log.error (/srv/www/node_modules/elasticsearch/src/lib/log.js:226:60) at checkRespForFailure (/srv/www/node_modules/elasticsearch/src/lib/transport.js:244:18) at HttpConnector.<anonymous> (/srv/www/node_modules/elasticsearch/src/lib/connectors/http.js:153:7) at ClientRequest.wrapper (/srv/www/node_modules/lodash/index.js:3095:19) at ClientRequest.emit (events.js:95:17) at CleartextStream.socketErrorListener (http.js:1548:9) at CleartextStream.emit (events.js:95:17) at SecurePair.<anonymous> (tls.js:1411:15) at SecurePair.emit (events.js:95:17) at SecurePair.error (tls.js:1018:27)
So my question is: why does the ES client, while connected to https://host:443 tries to connect to different hosts when invoking client.indices.putTemplate()? It is already connected to the cluster, why putting template makes the client reconnect on, what seems, bogus protocol/host/port combo?

Setup:

  • Elasticsearch cluster of 3 master nodes, 3 data nodes and 2 client nodes.
  • Elasticsearch.js (client set to version 1.5)
2

If I change my configuration to connect the ES client to plain http (http://:9200) everything works fine. If I connect using https://:443, I get the following error:

PUT https://:9200/_template/template_mapping?order=2&create=true&master_timeout=600000 => socket hang up

But... you're not connecting to port 443. It's still port 9200, and naturally ES isn't happy about you trying to use SSL on a non-SSL port.

So my question is: why does the ES client, while connected to https:// tries to connect to different hosts when invoking client.indices.putTemplate()

What hosts and ports are you talking about?

3

When I construct the ES client, I give it this host https://host:433 and it connects and functions fine.

For normal operations (searches and gets), the ES client (connected to https://host:443) works fine.
On occasion, we need to update the templates for indices. when the same client is used to put templates using client.indices.putTemplate, I get the error mentioned above.

If the ES client is constructed with http://host:9200, everything (searches, gets, putTemplates) works.

What I have a hard time understanding is while my client is constructed with https://host:443, why does it try to (re)connect to https://host:9200 when invoking putTemplate()?

4

Turned out that the javascript client was configured with sniffOnStart = true
and sniffOnConnectionFault = true which made it try to connect to internal cluster hosts that didn't support ssl. Ssl is handled by nginx not elasticsearch. Setting these 2 settings to false, made the client stop looking for other cluster hosts and continued using the existing ssl connection to nginx on port 443, which then forwards to plain http on 9200 to elasticsearch cluster.