Elasticsearch.js API request timeout

[dmongit]

Short story: I'm using the es.js client creation code for the api. I call bulk with the api. I get a request timeout. I'm passing a newline delimited json array.

Longer story (read the short story first, please):

My code is:

var client = new elasticsearch.Client({
  host: {
	protocol: 'http',
	host: 'myhost.com',
	path: '/myindex/_doc'
  },
  log: ['error', 'trace'],
  sniffOnStart: false,
  requestTimeout: 110000,
  pingTimeout: 110000,
  createNodeAgent(connection, config) { // TODO: bug, connection doesn't stay alive "no living connection"
	return new AgentKeepAlive(connection.makeAgentConfig(config));
  }
});

My bulk code:

client.bulk({
	body: out
  }, function (err, resp) {
	console.log('in');
	console.log(err);
	console.log(resp);
});

The out object is (abbrev):

[
'{"index":{"... }}
'{"myname":"value"}'
]

I do wonder if the single quotes around each array element is problematic, BUT, I'm using AWS, and AWS shows the ES trace:

-> POST http://myhost.com:9200/index/type/_bulk
{"index":{"...}}
{"myname":"value"}
<- 0

The ES Trace is written to the log after the client.bulk request timeout error... maybe a timing issue with async? I dunno... I'm grasping for an explanation.

Thanks for the help!

edit: I'm using 15.0.0 of es.js api.

[dmongit]

This is the debug log (I've added refresh: "wait_for")...

Elasticsearch DEBUG: 2018-06-27T22:48:34Z
starting request {
"method": "POST",
"bulkBody": true,
"path": "/_bulk",
"body": [
"{"index":{"_id":"afdasdfafds"}}",
"\n",
"{"...
"\n",
"{"index":{"_id":"asdadsfasdfasfd"}}",
"\n",
"{"...
"\n",
"{"index":{"_id":"asdfasdfasfdasdfafdasdf"}}",
"\n",
"{"...
"\n"
],
"query": {
"refresh": "wait_for"
}
}

[dmongit]

Update 2: If my hostname includes a .com (myhost.com) nothing happens... not even a client.ping shows it's been attempted. If I remove the .com (myhost) from the hostname, the ping and bulk make an attempt but there is a no living connection warning.

Why? I'm using an AWS endpoint, so I kind of need that .com, don't I?

[dadoonet (David Pilato)]

You are using elasticsearch service provided by AWS? If so you should ask their support.

BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is the only way to have access to X-Pack. Think about what is there yet like Security, Monitoring, Reporting and what is coming like Canvas, SQL...

[dmongit]

I am using ES hosted on AWS. However, I'm using the Elastic elasticsearch js API. That belongs to Elastic? An endpoint is an endpoint, right? It's almost like there is code that sees a competing cloud service and blocks the connection. My code connects to localhost:9200 but not an AWS endpoint. Conspiracy theory, maybe. Also, I don't think I need x-pack.

[Christian_Dahlqvist]

As far as I know AWS ES requires authentication, and I do not believe it supports HTTP auth and role-based access control like Elastic Cloud does. For Logstash they therefore developed a special Elasticsearch plugin that handles this. As I have no experience with AWS ES service I can not tell whether it is possible to configure elasticsearch.js to connect to it or not.

I you do not get any response from other users here I would recommend you reaching out to AWS support as they should know exactly what is required.

[dadoonet (David Pilato)]

@dmongit

It's almost like there is code that sees a competing cloud service and blocks the connection.

As the JS client is open source, you can read its code. I highly doubt you will see anything like that in the code.

[dmongit]

@dadoonet

As the JS client is open source, you can read its code. I highly doubt you will see anything like that in the code.

I was being facetious... my coping mechanism for this frustration. I know i'm super close and I've researched this for a bit. I'm probably a brain fart away from figuring it out... or finding the npm package to use where someone else figured it out for me.