ES Cluster via proxy IP, but listener timeout after waiting for [30000] ms

Elastic Stack Elasticsearch
1

Hi Everyone,

I ran into below issue.
My ES cluster has 3 nodes, and I setup one proxy IP, you can regard it as Loadbalance IP, which points to 3 nodes with ip_hash.

ES Version: 6.0.0

But when I call search API via es rest client, it throws below exception.

Something wrong with middleware: java.io.IOException: listener timeout after waiting for [30000] ms at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:661) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:220) at org.elasticsearch.client.RestClient.performRequest(RestClient.java:192) at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:428) at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:414) at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:355) at com.cargosmart.b2b.fwk2.process.bkt.dao.impl.BizkeyDaoESImpl.retrieveTransactionId(BizkeyDaoESImpl.java:152) at com.cargosmart.b2b.fwk2.process.bkt.controller.BizkeyTrackMiddleware.process(BizkeyTrackMiddleware.java:62) at org.apache.camel.processor.DelegateSyncProcessor.process(DelegateSyncProcessor.java:63) at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77) at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:541) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198) at org.apache.camel.processor.Pipeline.process(Pipeline.java:120) at org.apache.camel.processor.Pipeline.process(Pipeline.java:83) at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:198) at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:112)

I also tried it in server side.
For bigIP via telnet, it works:

[user@elk ~]$ curl http://bigIP:9200                                                  
 {
  "name" : "node1",
  "cluster_name" : "mycluster",
  "cluster_uuid" : "xt66V3oDTP2FsXj66eIWIQ",
  "version" : {
    "number" : "6.0.0",
    "build_hash" : "8f0685b",
    "build_date" : "2017-11-10T18:41:22.859Z",
    "build_snapshot" : false,
    "lucene_version" : "7.0.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

But stuck in below search API and without response:

[user@elk ~]$ curl -XGET 'http://bigIP:9200/_search?pretty' -H 'Content-Type: application/json' -d'{"query": {"bool": {"must": [{ "match": { "title":"Search"}}, { "match": { "content": "Elasticsearch" }}],"filter": [ {"term":  { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ]}}}'

Per trying in one es server node by curl, all nodes are running normally

[user@elk ~]$ curl -XGET 'http://node1:9200/_search?pretty' -H 'Content-Type: application/json' -d'{"query": {"bool": {"must": [{ "match": { "title":"Search"}}, { "match": { "content": "Elasticsearch" }}],"filter": [ {"term":  { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ]}}}'          
 {
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}

Very very strange, I can search full record via bigip:9200 in command line based:

[user@elk ~]$  curl -XGET http://bigIP:9200/_search?pretty

However not for below one:

[user@elk ~]$ curl -XGET 'http://bigIP:9200/_search?pretty' -H 'Content-Type: application/json' -d'{"query": {"bool": {"must": [{ "match": { "title":"Search"}}, { "match": { "content": "Elasticsearch" }}],"filter": [ {"term":  { "status": "published" }}, { "range": { "publish_date": { "gte": "2015-01-01" }}} ]}}}'

Anyone has the same issue? Please help me jump out of this damn thing. Thanks advanced.

Something I can confirm is as below:

  • Not rest API version compatability issue.
2

This might be a limitation of your proxy. It might not support GET with body.
Try to run the same request with POST instead.

3

@dadoonet, sounds a good advise. Let's jump into the proxy server setting to dig it out.

4

Hi @dadoonet , may I know, at es rest client side, if call es with curl -XGET backend?

5

I don't understand your question. Could you rephrase it?

6

Means that if we use Java rest client to search es result, like:

SearchRequest searchRequest = new SearchRequest(); 
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder(); 
searchSourceBuilder.query(QueryBuilders.matchAllQuery());

If request with curl -XGET which java rest client used? Thanks.

7

We are using GET indeed. I just checked the code.

I opened this

For discussion with the team.

8

Thanks.

9

Could you confirm that using curl with GET does not work but curl with POST works?

And also, what is the proxy you are using?

10 
# hung without response
[user@elk~]$ curl -XGET 'http://bigIP:9200/_search?pretty' -H 'Content-Type: application/json' -d'
> {
>   "query": {
>     "bool": {
>       "must": [
>         { "match": { "title":   "Search"        }},
>         { "match": { "content": "Elasticsearch" }}
>       ],
>       "filter": [
>         { "term":  { "status": "published" }},
>         { "range": { "publish_date": { "gte": "2015-01-01" }}}
>       ]
>     }
>   }
> }
> '

^C

# response with result
[user@elk ~]$ curl -XPOST 'http://bigIP:9200/_search?pretty' -H 'Content-Type: application/json' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "title":   "Search"        }},
        { "match": { "content": "Elasticsearch" }}
      ],
      "filter": [
        { "term":  { "status": "published" }},
        { "range": { "publish_date": { "gte": "2015-01-01" }}}
      ]
    }
  }
}
'
{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 6,
    "successful" : 6,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 0,
    "max_score" : null,
    "hits" : [ ]
  }
}
[user@elk ~]$

I use F5 BIG IP.

11

Thanks! We will fix this unless you are a Java dev and you want to propose a PR. :stuck_out_tongue:

12

You expert dev are preferred. :grinning:

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.