Display more than 10k records in sense plugin and elasticsearch-sql plugin

[sistla (divya sistla)]

Hi,

We are using Elasticsearch 2.4.6 , logstash 2.2 and kibana 4.1.6 and installed two plugins to query in the elastic search . But we are able to display only 10,000 records among the existing 1 million records .

Here is our code ,

GET /good/_search?size=10000
{
"query" : {
"bool":{
"must":[
{"match": { "Drop_EccbTmoRrcConnReEstb": "4" }}
]
}
}
}

[dadoonet (David Pilato)]

Elasticsearch limits by default the number of records you can get back from a search request to 10000.
If you want more than that, call the scroll API or look at the search_after feature.

[sistla (divya sistla)]

We tried scroll API and search after feature , but its not showing more than 10k records . Please let us know what to do

Our code :

GET /good/_search?scroll=1m
{
"query" : {
"term" : { "eCSFB_to_CDMA1XRTT_Exe_Succ_Count" : 24 }
},
"filter":{
"and" : [
{
"term" : { "eCSFB_to_CDMA1XRTT_Prep_Succ_Count" : 24 }
}

        ]
   }

}

[dadoonet (David Pilato)]

What is happening when you call the scroll API with the scroll ID that was given to you after the first call?

[sistla (divya sistla)]

code:

GET /good/_search?scroll
{
"scroll" : "1m",
"_scroll_id": "cXVlcnlUaGVuRmV0Y2g7NTsyNDM5MTk6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjE6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjA6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjI6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjM6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzswOw==",
"query" : {
"term" : { "eCSFB_to_CDMA1XRTT_Exe_Succ_Count" : 24 }
},
"filter":{
"and" : [
{
"term" : { "eCSFB_to_CDMA1XRTT_Prep_Succ_Count" : 24 }
}

        ]
   }

}

output:
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "Failed to parse setting [scroll] with value [] as a time value: unit is missing or unrecognized"
}
],
"type": "parse_exception",
"reason": "Failed to parse setting [scroll] with value [] as a time value: unit is missing or unrecognized"
},
"status": 400
}

[dadoonet (David Pilato)]

Scroll API does not work as you posted. See https://www.elastic.co/guide/en/elasticsearch/reference/6.3/search-request-scroll.html

Second call should be

POST  /_search/scroll 
{
    "scroll" : "1m", 
    "scroll_id" : "cXVlcnlUaGVuRmV0Y2g7NTsyNDM5MTk6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjE6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjA6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjI6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzsyNDM5MjM6WE1pZml1N2RSaXU1Q2V3cU1aMF90ZzswOw==" 
}