Elasticsearch query time range issue

dadoonet · April 8, 2021, 9:25am

Could start again from the beginning? When there is no pit yet?

Something like:

POST /INDEXNAME/_pit?keep_alive=5m

And then share here the request AND the response.

Then run:

POST /_search 
{
   "size":1,
   "_source":[
      "kubernetes.container.name",
      "kubernetes.namespace",
      "message",
      "@timestamp"
   ],
   "query":{
      "bool":{
         "must":{
            "match":{
               "message":"exception"
            }
         },
         "filter":{
            "range":{
               "@timestamp":{
                  "gte":"2021-04-06T01:00:00",
                  "lte":"2021-04-06T18:00:00"
               }
            }
         }
      }
   },
   "pit":{
      "id":"PIT_ID_THAT_YOU_GOT",
      "keep_alive":"5m"
   },
   "sort":[
      {
         "@timestamp":"asc"
      }
   ]
}

And share here the request AND the result. Note that I set size: 1 so the result will be small enough to be pasted here.

suresh123 · April 8, 2021, 9:53am

Hi David,

PIT ID
{"id":"48myAwcaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDgWY1d0MzhzSERRZnF6eGNNLVJubko0QQAWWWNmSml6aVdRR21yNFJpTWZ5U3FCdwAAAAAAAA0paBZveVVmVFRUM1JwVzFOcXpvSF9ITW5BABpkZXYyLWFwcGxvZ3MtYmUtMjAyMS4wNC4wNRZqWlAyYXdwSVRTS1hqRUh4bGI2Uk93ABZjUDZQYl9zcVJsQ0h1Sk5JS2RaRERnAAAAAAAAAmldFm9Ec1FORjQyUXZpam5VYTB4ekh2Y3cAGmRldjItYXBwbG9ncy1iZS0yMDIxLjA0LjA3FmFEQWotRURYU2pHbjVBdm1mTWtSNGcAFlljZkppemlXUUdtcjRSaU1meVNxQncAAAAAAAANKVwWb3lVZlRUVDNScFcxTnF6b0hfSE1uQQAaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDIWTmNudXJCdS1UMldVQWtHcnhyR3NDZwAWY1A2UGJfc3FSbENIdUpOSUtkWkREZwAAAAAAAAJpXBZvRHNRTkY0MlF2aWpuVWEweHpIdmN3ABpkZXYyLWFwcGxvZ3MtYmUtMjAyMS4wNC4wNhZDUGRfUF9RMVJEdVMxYmpLNlViSWJBABZFamdFTTBoaFIyV0RzMF9DbkcycFFRAAAAAAAAAe06FnVVazE1dXRWVDJDTmdKWFUtekNRMmcAGmRldjItYXBwbG9ncy1iZS0yMDIxLjA0LjA0FkRNUUpHd1l3VHlLbjVJM0JOSlZRSGcAFkVqZ0VNMGhoUjJXRHMwX0NuRzJwUVEAAAAAAAAB7TkWdVVrMTV1dFZUMkNOZ0pYVS16Q1EyZwAaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDMWZlhWT2FjR3hRRktFSDlhekNZNlBZQQAWRWpnRU0waGhSMldEczBfQ25HMnBRUQAAAAAAAAHtOBZ1VWsxNXV0VlQyQ05nSlhVLXpDUTJnAAcWRE1RSkd3WXdUeUtuNUkzQk5KVlFIZwAAFmNXdDM4c0hEUWZxenhjTS1Sbm5KNEEAABZmWFZPYWNHeFFGS0VIOWF6Q1k2UFlBAAAWQ1BkX1BfUTFSRHVTMWJqSzZVYkliQQAAFmFEQWotRURYU2pHbjVBdm1mTWtSNGcAABZqWlAyYXdwSVRTS1hqRUh4bGI2Uk93AAAWTmNudXJCdS1UMldVQWtHcnhyR3NDZwAA"}

Still same error

"root_cause" : [
{
"type" : "search_context_missing_exception",
"reason" : "No search context found for id [158044]"
},

POST /_search 
{
   "size":1,
   "_source":[
      "kubernetes.container.name",
      "kubernetes.namespace",
      "message",
      "@timestamp"
   ],
   "query":{
      "bool":{
         "must":{
            "match":{
               "message":"exception"
            }
         },
         "filter":{
            "range":{
               "@timestamp":{
                  "gte":"2021-04-06T01:00:00",
                  "lte":"2021-04-06T18:00:00"
               }
            }
         }
      }
   },
   "pit":{
      "id":"48myAwcaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDgWY1d0MzhzSERRZnF6eGNNLVJubko0QQAWWWNmSml6aVdRR21yNFJpTWZ5U3FCdwAAAAAAAA0paBZveVVmVFRUM1JwVzFOcXpvSF9ITW5BABpkZXYyLWFwcGxvZ3MtYmUtMjAyMS4wNC4wNRZqWlAyYXdwSVRTS1hqRUh4bGI2Uk93ABZjUDZQYl9zcVJsQ0h1Sk5JS2RaRERnAAAAAAAAAmldFm9Ec1FORjQyUXZpam5VYTB4ekh2Y3cAGmRldjItYXBwbG9ncy1iZS0yMDIxLjA0LjA3FmFEQWotRURYU2pHbjVBdm1mTWtSNGcAFlljZkppemlXUUdtcjRSaU1meVNxQncAAAAAAAANKVwWb3lVZlRUVDNScFcxTnF6b0hfSE1uQQAaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDIWTmNudXJCdS1UMldVQWtHcnhyR3NDZwAWY1A2UGJfc3FSbENIdUpOSUtkWkREZwAAAAAAAAJpXBZvRHNRTkY0MlF2aWpuVWEweHpIdmN3ABpkZXYyLWFwcGxvZ3MtYmUtMjAyMS4wNC4wNhZDUGRfUF9RMVJEdVMxYmpLNlViSWJBABZFamdFTTBoaFIyV0RzMF9DbkcycFFRAAAAAAAAAe06FnVVazE1dXRWVDJDTmdKWFUtekNRMmcAGmRldjItYXBwbG9ncy1iZS0yMDIxLjA0LjA0FkRNUUpHd1l3VHlLbjVJM0JOSlZRSGcAFkVqZ0VNMGhoUjJXRHMwX0NuRzJwUVEAAAAAAAAB7TkWdVVrMTV1dFZUMkNOZ0pYVS16Q1EyZwAaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDMWZlhWT2FjR3hRRktFSDlhekNZNlBZQQAWRWpnRU0waGhSMldEczBfQ25HMnBRUQAAAAAAAAHtOBZ1VWsxNXV0VlQyQ05nSlhVLXpDUTJnAAcWRE1RSkd3WXdUeUtuNUkzQk5KVlFIZwAAFmNXdDM4c0hEUWZxenhjTS1Sbm5KNEEAABZmWFZPYWNHeFFGS0VIOWF6Q1k2UFlBAAAWQ1BkX1BfUTFSRHVTMWJqSzZVYkliQQAAFmFEQWotRURYU2pHbjVBdm1mTWtSNGcAABZqWlAyYXdwSVRTS1hqRUh4bGI2Uk93AAAWTmNudXJCdS1UMldVQWtHcnhyR3NDZwAA",
      "keep_alive":"5m"
   },
   "sort":[
      {
         "@timestamp":"asc"
      }
   ]
}

suresh123 · April 8, 2021, 10:07am

I executed same thing in devtools, working fine.

I need to do same thing in server.

{
"pit_id" : "48myAwcaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDgWY1d0MzhzSERRZnF6eGNNLVJubko0QQAWWWNmSml6aVdRR21yNFJpTWZ5U3FCdwAAAAAAAA01GxZveVVmVFRUM1JwVzFOcXpvSF9ITW5BABpkZXYyLWFwcGxvZ3MtYmUtMjAyMS4wNC4wNRZqWlAyYXdwSVRTS1hqRUh4bGI2Uk93ABZZY2ZKaXppV1FHbXI0UmlNZnlTcUJ3AAAAAAAADTUZFm95VWZUVFQzUnBXMU5xem9IX0hNbkEAGmRldjItYXBwbG9ncy1iZS0yMDIxLjA0LjA3FmFEQWotRURYU2pHbjVBdm1mTWtSNGcAFmNQNlBiX3NxUmxDSHVKTklLZFpERGcAAAAAAAACcvcWb0RzUU5GNDJRdmlqblVhMHh6SHZjdwAaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDIWTmNudXJCdS1UMldVQWtHcnhyR3NDZwAWWWNmSml6aVdRR21yNFJpTWZ5U3FCdwAAAAAAAA01GBZveVVmVFRUM1JwVzFOcXpvSF9ITW5BABpkZXYyLWFwcGxvZ3MtYmUtMjAyMS4wNC4wNhZDUGRfUF9RMVJEdVMxYmpLNlViSWJBABZjUDZQYl9zcVJsQ0h1Sk5JS2RaRERnAAAAAAAAAnL2Fm9Ec1FORjQyUXZpam5VYTB4ekh2Y3cAGmRldjItYXBwbG9ncy1iZS0yMDIxLjA0LjA0FkRNUUpHd1l3VHlLbjVJM0JOSlZRSGcAFkVqZ0VNMGhoUjJXRHMwX0NuRzJwUVEAAAAAAAAB8GkWdVVrMTV1dFZUMkNOZ0pYVS16Q1EyZwAaZGV2Mi1hcHBsb2dzLWJlLTIwMjEuMDQuMDMWZlhWT2FjR3hRRktFSDlhekNZNlBZQQAWRWpnRU0waGhSMldEczBfQ25HMnBRUQAAAAAAAAHwaBZ1VWsxNXV0VlQyQ05nSlhVLXpDUTJnAAcWRE1RSkd3WXdUeUtuNUkzQk5KVlFIZwAAFmNXdDM4c0hEUWZxenhjTS1Sbm5KNEEAABZmWFZPYWNHeFFGS0VIOWF6Q1k2UFlBAAAWQ1BkX1BfUTFSRHVTMWJqSzZVYkliQQAAFmFEQWotRURYU2pHbjVBdm1mTWtSNGcAABZqWlAyYXdwSVRTS1hqRUh4bGI2Uk93AAAWTmNudXJCdS1UMldVQWtHcnhyR3NDZwAA",
"took" : 54,
"timed_out" : false,
"_shards" : {
"total" : 7,
"successful" : 7,
"skipped" : 6,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 88,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "dev2-applogs-be-2021.04.06",
"_type" : "_doc",
"_id" : "GZrJpXgBm5biRIvY8df6",
"_score" : null,
"_source" : {
"kubernetes" : {
"container" : {
"name" : "hp-wcc-services-search"
},
"namespace" : "dev2"
},
"@timestamp" : "2021-04-06T06:06:11.335Z",
"message" : """2021-04-06 06:06:11.332 WARN [search-service,,,] Check result of the [AsyncReporter{org.springframework.cloud.sleuth.zipkin2.sender.RestTemplateSender@41be8db9}] contains an error [CheckResult{ok=false, error=org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://wcc-monitor.corp.hpicloud.net:9411/api/v2/spans": connect timed out; nested exception is java.net.SocketTimeoutException: connect timed out}]"""
},
"sort" : [
1617689171335,
48550
]
}
]
}
}

dadoonet · April 8, 2021, 10:22am

Please read again:

And:

Please format ALL the code parts as you did for only the last part of the code in this answer Elasticsearch query time range issue - #43 by suresh123.

suresh123 · April 8, 2021, 10:23am

Hi David,

It woks perfectly. Let me get more then 1 lakh count.

Thanks

suresh123 · April 8, 2021, 11:01am

Hi David,

I am getting sort for every request

"sort" : [
1617731997062,
223299
]

So, which sort numbers I need to add in search_after ?
"search_after": [
4098435132000,
4294967298
],

dadoonet · April 8, 2021, 11:26am

Both I guess.

suresh123 · April 8, 2021, 11:35am

There are thousands of sorts, which sort numbers i need to give exactly
Example

{
        "_index" : "dev-nginx-logs-2021.04.08",
        "_type" : "_doc",
        "_id" : "ROTPrngBm5biRIvY1qF3",
        "_score" : null,
        "_source" : {
          "request" : "image1",
          "referrer" : "-",
          "response_code" : 404,
          "@timestamp" : "2021-04-08T00:09:19.193Z"
        },
        "sort" : [
          1617840559193,
          4294968280
        ]
      },
      {
        "_index" : "dev-nginx-logs-2021.04.08",
        "_type" : "_doc",
        "_id" : "KuvPrngB9h4iHZ0t9DoD",
        "_score" : null,
        "_source" : {
          "request" : "/images/windows_image",
          "referrer" : "-",
          "response_code" : 404,
          "@timestamp" : "2021-04-08T00:09:26.194Z"
        },
        "sort" : [
          1617840566194,
          4294968281
        ]
      },

This is the search_after which sort I need to give.
"search_after": [
4098435132000,
4294967298
],

suresh123 · April 8, 2021, 12:17pm

Hi David,

I took last value as it is giving Ascending order.

Thanks

dadoonet · April 8, 2021, 12:27pm

I think everything is good. So we can close this long thread now.

Feel free to open a new question if needed.

suresh123 · April 8, 2021, 12:41pm

Thanks you somuch.

system · May 6, 2021, 12:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.