Unstable multi search response

[honey.and.sw]

Hi community!

I have one Elasticsearch node, 8 vCPU and 30G mem machine, 10G is allocated for heap.
The problem is multi search by Java SDK fails quite often and its response alternates success and failure.

When it fails I can see this error from Java SDK:

org.elasticsearch.transport.NodeNotConnectedException: (...) Node not connected
	at org.elasticsearch.transport.ConnectionManager.getConnection(ConnectionManager.java:166) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:587) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:559) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.action.TransportActionNodeProxy.execute(TransportActionNodeProxy.java:50) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.client.transport.TransportProxyClient.lambda$execute$0(TransportProxyClient.java:60) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:252) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:382) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:395) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:384) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) ~[elasticsearch-6.5.1.jar:6.5.1]
	at org.elasticsearch.action.ActionRequestBuilder.get(ActionRequestBuilder.java:53) ~[elasticsearch-6.5.1.jar:6.5.1]

My codes are the below in Scala. What it does is to get totalHit of each date range so I pass size as 0.

    // main part
    val msearch = client.prepareMultiSearch()

    ranges.map { case (gte, lte) =>
      val query = buildTweetsDateRangeQuery(gte, lte, includeRt)
      buildSearchRequest(query, Some(0), None)
    }.foreach(msearch.add)

    msearch.get()

  private def buildSearchRequest(query: QueryBuilder,
                                 size: Option[Int],
                                 searchAfter: Option[Array[AnyRef]]): SearchRequestBuilder = {
    val builder = client.prepareSearch("twitter")
      .setSearchType(SearchType.DEFAULT)
      .setQuery(query)
      .addSort(DEFAULT_SEARCH_SORT)
      .setSize(size.getOrElse(DEFAULT_SIZE))

    searchAfter.foreach(builder.searchAfter)

    builder
  }

If I send _msearch via curl it seems fine. The difference is it doesn't limit size to 0 but I cannot see this makes big impact (maybe I'm wrong).

{"index": "twitter"}
{"query" : {"range" : {"createdAt" : {"gte" : 1296169200000, "lte" : 1296687600000}}}}
{"index": "twitter"}
{"query" : {"range" : {"createdAt" : {"gte" : 1327705200000, "lte" : 1328223600000}}}}
...

Because there is no error log on server side, it is hard to debug it.
Hope I can get some insight!