Not able to retrieve doc using transport client but works through REST

Elastic Stack Elasticsearch
1

Hi

Initially we were using ES 2.4, recently upgraded to ES 6.6, but facing an issue with specific document type.

We have a index with name logs-2019.05.28.14 have multiple types of data, but for simplicity we had kept one type of data and tested

sample of mappings are given below

{
  "order": 50, 
  "index_patterns": ["logs-*"], 
  "mappings": {
    "log": {
      "_all": {
        "enabled": false
      }, 
      "properties": {
  	    "metricType": {
          "type": "keyword"
        } .,
    "fwd_zero_wnd": {
      "type": "long", 
      "doc_values": false
    }, 
    "custom2_name": {
      "norms": false,  
      "type": "keyword", 
      "index_options": "docs"
    },

And we use transport client 6.6.0. We fire a query using client and we get empty hits. But when we take tostring of request query and execute exactly same using REST it returns the result

When we took tcp dump and see, as below

06:38:12.386690 IP 192.168.13.22.42760 > 192.168.13.13.9300: Flags [P.], seq 120:906, ack 121, 
win 218, options [nop,nop,TS val 470802143 ecr 470799447], length 786

..$T...4E.2h...........

.......WES..............\xC....transport_client.indices:data/read/search....logs-2019.05.28.05.logs- 
2019.05.28.06.logs-2019.05.28.07.logs-2019.05.28.08.logs-2019.05.28.09.logs- 
2019.05.28.10.logs-2019.05.28.11.logs-2019.05.28.12.logs-2019.05.28.13.logs- 
2019.05.28.14.logs-2019.05.28.15.logs-2019.05.28.16.logs-2019.05.28.17.logs- 
2019.05.28.18.logs-2019.05.28.19.logs-2019.05.28.20.logs-2019.05.28.21.logs- 
2019.05.28.21.errors-2019.05.28.errors- 
2019.05.28................bool?......range?.....ts....j..0.....k.r.J......match?....
 
metricType. cgn_l4_pc..2..........bool?........bool?........match?.... accountId.$fd457d5e-7ef9-11e9- 
 9cd1-6a9aaf215cbe..2.............bool?........match?....

account_id.$fd457d5e-7ef9-11e9-9cd1-6a9aaf215cbe..2................1.......................logs.............

06:38:12.398964 IP 192.168.13.13.9300 > 192.168.13.22.42760: Flags [P.], seq 121:167, ack 906, 
win 220, options [nop,nop,TS val 470802156 ecr 470802143], length 46

.$T..E.2h...F...........

........ES...(..........\xC.........................


06:38:12.399036 IP 192.168.13.22.42760 > 192.168.13.13.9300: Flags [.], ack 167, win 218, options 
  [nop,nop,TS val 470802156 ecr 470802156], length 0

..$T...FE.2............

........

06:38:14.686521 IP 192.168.13.22.42774 > 192.168.13.13.9300: Flags [P.], seq 120:126, ack 121, 
win 209, options [nop,nop,TS val 470804443 ecr 470799444], length 6

..$T++....O............

.......TES....

And when we look at logs there are no exception or anything

And we have enabled trace and output is as below

       [2019-05-28T07:18:13,143][TRACE][o.e.t.TransportLogger    ] [metrics-datastore-0] Netty4TcpChannel{localAddress=/192.168.13.13:9300, remoteAddress=/192.168.13.22:42760}   [length: 786, request id: 1386, type: request, version: 6.6.0, action: indices:data/read/search] READ: 786B
  [2019-05-28T07:18:13,143][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [1386][indices:data/read/search] received request
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39087][indices:data/read/search[phase/query]] sent to [{metrics-datastore-0}{dJ4qfA01SUidhL51vWsg2g}{PHzNYxFbSyC4OlqFuz9row}{192.168.13.13}{192.168.13.13:9300}{ml.machine_memory=126569361408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] (timeout: [null])
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39087][indices:data/read/search[phase/query]] received request
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39088][indices:data/read/search[phase/query]] sent to [{metrics-datastore-0}{dJ4qfA01SUidhL51vWsg2g}{PHzNYxFbSyC4OlqFuz9row}{192.168.13.13}{192.168.13.13:9300}{ml.machine_memory=126569361408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] (timeout: [null])
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39088][indices:data/read/search[phase/query]] received request
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39089][indices:data/read/search[phase/query]] sent to [{metrics-datastore-0}{dJ4qfA01SUidhL51vWsg2g}{PHzNYxFbSyC4OlqFuz9row}{192.168.13.13}{192.168.13.13:9300}{ml.machine_memory=126569361408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] (timeout: [null])
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39089][indices:data/read/search[phase/query]] received request
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39090][indices:data/read/search[phase/query]] sent to [{metrics-datastore-0}{dJ4qfA01SUidhL51vWsg2g}{PHzNYxFbSyC4OlqFuz9row}{192.168.13.13}{192.168.13.13:9300}{ml.machine_memory=126569361408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] (timeout: [null])
  [2019-05-28T07:18:13,144][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39090][indices:data/read/search[phase/query]] received request
  [2019-05-28T07:18:13,145][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39091][indices:data/read/search[phase/query]] sent to [{metrics-datastore-0}{dJ4qfA01SUidhL51vWsg2g}{PHzNYxFbSyC4OlqFuz9row}{192.168.13.13}{192.168.13.13:9300}{ml.machine_memory=126569361408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}] (timeout: [null])
  [2019-05-28T07:18:13,145][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39091][indices:data/read/search[phase/query]] received request
  [2019-05-28T07:18:13,146][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [metrics-datastore-0] [logs-2019.05.28.14] not opting out of the query cache; authorization is not allowed
  [2019-05-28T07:18:13,146][DEBUG][o.e.x.s.a.a.OptOutQueryCache] [metrics-datastore-0] [logs-2019.05.28.14] not opting out of the query cache; authorization is not allowed
  [2019-05-28T07:18:13,147][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39087][indices:data/read/search[phase/query]] sent response
  [2019-05-28T07:18:13,147][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39087][indices:data/read/search[phase/query]] received response from [{metrics-datastore-0}{dJ4qfA01SUidhL51vWsg2g}{PHzNYxFbSyC4OlqFuz9row}{192.168.13.13}{192.168.13.13:9300}{ml.machine_memory=126569361408, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}]
  [2019-05-28T07:18:13,147][TRACE][o.e.t.T.tracer           ] [metrics-datastore-0] [39088][indices:data/read/search[phase/query]] sent response
2

REST request is

curl -XPOST 'localhost:9200/logs-2019.05.28.14/_search?pretty' -H 'Content-Type: application/json' -d '
{
      "from": 0,
      "size": 1,
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "ts": {
              "from": 1559002627103,
              "to": 1559092627103,
              "include_lower": true,
              "include_upper": true,
              "boost": 1.0
            }
          }
        }
      ]
    }
  }
}
'

and i get the result

3

can you share the java code snippet as well?

4

Hi i will not be able to share exact code because its spread across files. But here is the below idea

private TransportClient createTransportClient() {
	Settings.Builder settings = Settings.builder().put("cluster.name", appConfiguration.getDSClusterName())
			.put("client.transport.sniff", false)
			.put("client.transport.ping_timeout", appConfiguration.getPingTimeout());
	return new PreBuiltTransportClient(settings.build()); // - UpgradeChange
}

 IndicesOptions FROM_OPTIONS = IndicesOptions.fromOptions(true, true, true, true);

SearchRequestBuilder setQuery = dataStoreConnection.getTransportClient().prepareSearch().setIndices(indexName)
			.setIndicesOptions(FROM_OPTIONS).setTypes(mappingName).setQuery(filterBuilder).setFrom(from);
5

Can anybody help me how can i get whats happening at elasticsearch wrt that document

6

Can you use the exact same index options in the rest request as well and see what happens?

Right now those search requests seem pretty different to me.

7

the rest request i pasted is here exactly the one produced by java api, its the toString() on searchRequest.

8

what about the indices options in your rest request?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.