Array index out of bound with scroll search

[Matias_Antinanco (Matias Antiñanco)]

Hi,

we have an ES cluster with 18 nodes: 15 data nodes and 3 master nodes. The cluster has around 750 indexes with 7 shards each, we are doing some scan&scroll searches to paginate the results and the bigger the search criteria is (and therefore the amount of shards impacted) the longer the scrollId is.

Now the problem is that when scrollId is too long we are getting an array_index_out_of_bounds_exception.

We are executing the queries in a java application using spring restTemplate but the error also happens sending the request via curl. Basically, a POST or GET request with an scrollId from around 4000 chars length works fine:

curl -XGET 'IP:9200/_search/scroll' -H 'Content-Type: application/json' -d'
{
"scroll" : "1m",
"scroll_id" : "cXVlcnlUaGVuRmV0Y2g7MTEyOzI1NzQ6aEtKN0tzanlTYkd0bWR3cFJUYzlMQTs2MjI6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs2MjM6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs0NDM6NGp5WWhTSlpUMHlSVjEtb3NPQXM1UTs2MjU6YlliRHBGRldSY0dicmM5WExVQ1J3UTs2MjY6VVZ2Wmp6VGJTWWVubFVpbTRiRWZHZzs2Mjc6VVZ2Wmp6VGJTWWVubFVpbTRiRWZHZzs2Mjg6VVZ2Wmp6VGJTWWVubFVpbTRiRWZHZzs0OTg6TE13amx6cjdTSGVLN3JEY3NGQlg3ZzszODc4OnIwNGVWa1NGUS1DenQ0M0JaOGZOR3c7NjI0OnBFbWxwMUxoUkFXMmZ0dldVbDl5bmc7NjI1OnBFbWxwMUxoUkFXMmZ0dldVbDl5bmc7NjI2OnBFbWxwMUxoUkFXMmZ0dldVbDl5bmc7MjUzOko2M3JHWXZpVHl5eExyTXFfbzlOcWc7NjI5OlVWdlpqelRiU1llbmxVaW00YkVmR2c7NjMwOlVWdlpqelRiU1llbmxVaW00YkVmR2c7MTQzOmE2dXpGQ3g2U3B5MzZIeW1uQnJXd3c7NjMxOlVWdlpqelRiU1llbmxVaW00YkVmR2c7NjMyOlVWdlpqelRiU1llbmxVaW00YkVmR2c7NjMzOlVWdlpqelRiU1llbmxVaW00YkVmR2c7NDM0OnY3bXhzMjg2UTI2c0J1TUVscl9lcHc7NjM0OlVWdlpqelRiU1llbmxVaW00YkVmR2c7Mzg3OTpyMDRlVmtTRlEtQ3p0NDNCWjhmTkd3OzYzNTpVVnZaanpUYlNZZW5sVWltNGJFZkdnOzYzNjpVVnZaanpUYlNZZW5sVWltNGJFZkdnOzYyNzpwRW1scDFMaFJBVzJmdHZXVWw5eW5nOzM4MDotSWFsakVZZ1FUMnh1RkxnVEJ3bVRBOzM4ODA6cjA0ZVZrU0ZRLUN6dDQzQlo4Zk5HdzsxNDQ6YTZ1ekZDeDZTcHkzNkh5bW5Ccld3dzsyNTQ6SjYzckdZdmlUeXl4THJNcV9vOU5xZzs2MjY6YlliRHBGRldSY0dicmM5WExVQ1J3UTsyNTc2OmhLSjdLc2p5U2JHdG1kd3BSVGM5TEE7NjI3OmJZYkRwRkZXUmNHYnJjOVhMVUNSd1E7NjI4OmJZYkRwRkZXUmNHYnJjOVhMVUNSd1E7Mzg4MTpyMDRlVmtTRlEtQ3p0NDNCWjhmTkd3OzYzNzpVVnZaanpUYlNZZW5sVWltNGJFZkdnOzYyOTpiWWJEcEZGV1JjR2JyYzlYTFVDUndROzYzMDpiWWJEcEZGV1JjR2JyYzlYTFVDUndROzYzODpVVnZaanpUYlNZZW5sVWltNGJFZkdnOzYzMTpiWWJEcEZGV1JjR2JyYzlYTFVDUndROzYzMjpiWWJEcEZGV1JjR2JyYzlYTFVDUndROzYzMzpiWWJEcEZGV1JjR2JyYzlYTFVDUndROzYyMDo0UnE5WTF3aVNqLWFJdHJUUWRVSWdBOzI1NTpKNjNyR1l2aVR5eXhMck1xX285TnFnOzM4ODI6cjA0ZVZrU0ZRLUN6dDQzQlo4Zk5Hdzs2MjE6NFJxOVkxd2lTai1hSXRyVFFkVUlnQTs5NTQ6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTs0MTY6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzs0MTc6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzs2Mjg6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs0NTU6Y0ZBdF9MblBUZi0ydUNmNDZlTEZ5UTs0NTY6Y0ZBdF9MblBUZi0ydUNmNDZlTEZ5UTs2Mjk6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs5NTU6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTs5NTY6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTs5NTc6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTs5NTg6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTs5NTk6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTs5NjA6LVRjUUJTY0tTRG11bGJLVFl1YjJCQTszODgzOnIwNGVWa1NGUS1DenQ0M0JaOGZOR3c7NjM5OlVWdlpqelRiU1llbmxVaW00YkVmR2c7NjQwOlVWdlpqelRiU1llbmxVaW00YkVmR2c7MjU3NTpoS0o3S3NqeVNiR3RtZHdwUlRjOUxBOzM4MTotSWFsakVZZ1FUMnh1RkxnVEJ3bVRBOzY0MTpVVnZaanpUYlNZZW5sVWltNGJFZkdnOzQxODpOLXNxVXF3QVFNYXAzeXktUUJvTnJ3OzYyMjo0UnE5WTF3aVNqLWFJdHJUUWRVSWdBOzM4ODU6cjA0ZVZrU0ZRLUN6dDQzQlo4Zk5Hdzs2MjM6NFJxOVkxd2lTai1hSXRyVFFkVUlnQTs2MjQ6NFJxOVkxd2lTai1hSXRyVFFkVUlnQTs0MTk6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzszODI6LUlhbGpFWWdRVDJ4dUZMZ1RCd21UQTs0NTc6Y0ZBdF9MblBUZi0ydUNmNDZlTEZ5UTszODg0OnIwNGVWa1NGUS1DenQ0M0JaOGZOR3c7NjQyOlVWdlpqelRiU1llbmxVaW00YkVmR2c7MzgzOi1JYWxqRVlnUVQyeHVGTGdUQndtVEE7NjMwOnBFbWxwMUxoUkFXMmZ0dldVbDl5bmc7Mzg0Oi1JYWxqRVlnUVQyeHVGTGdUQndtVEE7MTQ1OmE2dXpGQ3g2U3B5MzZIeW1uQnJXd3c7NDk5OkxNd2psenI3U0hlSzdyRGNzRkJYN2c7Mzg4NjpyMDRlVmtTRlEtQ3p0NDNCWjhmTkd3OzY0MzpVVnZaanpUYlNZZW5sVWltNGJFZkdnOzYzNDpiWWJEcEZGV1JjR2JyYzlYTFVDUndROzUwMDpMTXdqbHpyN1NIZUs3ckRjc0ZCWDdnOzQyMDpOLXNxVXF3QVFNYXAzeXktUUJvTnJ3OzUwMTpMTXdqbHpyN1NIZUs3ckRjc0ZCWDdnOzUwMjpMTXdqbHpyN1NIZUs3ckRjc0ZCWDdnOzM4ODc6cjA0ZVZrU0ZRLUN6dDQzQlo4Zk5Hdzs0NTg6Y0ZBdF9MblBUZi0ydUNmNDZlTEZ5UTs1MDM6TE13amx6cjdTSGVLN3JEY3NGQlg3Zzs1MDQ6TE13amx6cjdTSGVLN3JEY3NGQlg3Zzs0MzU6djdteHMyODZRMjZzQnVNRWxyX2Vwdzs1MDU6TE13amx6cjdTSGVLN3JEY3NGQlg3Zzs0NDQ6NGp5WWhTSlpUMHlSVjEtb3NPQXM1UTs1MDY6TE13amx6cjdTSGVLN3JEY3NGQlg3ZzsyNTY6SjYzckdZdmlUeXl4THJNcV9vOU5xZzs2MzU6YlliRHBGRldSY0dicmM5WExVQ1J3UTs1MDc6TE13amx6cjdTSGVLN3JEY3NGQlg3Zzs0NDU6NGp5WWhTSlpUMHlSVjEtb3NPQXM1UTs0NTk6Y0ZBdF9MblBUZi0ydUNmNDZlTEZ5UTs2MzE6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs2MzI6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs0NjA6Y0ZBdF9MblBUZi0ydUNmNDZlTEZ5UTs2MzM6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs2MzQ6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs0MjE6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzs0MjI6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzs0MjQ6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzs2MzU6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs0MjM6Ti1zcVVxd0FRTWFwM3l5LVFCb05ydzs2MzY6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzs2Mzc6cEVtbHAxTGhSQVcyZnR2V1VsOXluZzswOw"
}
'

But with a longer scrollId (we are getting scrollIds from around 9000 chars) we are getting following error:

"error" : {
"root_cause" : [ {
"type" : "array_index_out_of_bounds_exception",
"reason" : "267"
} ],
"type" : "array_index_out_of_bounds_exception",
"reason" : "267"
},
"status" : 500
}

We are using ES 2.2.1, at the very beginning I thought it was a limitation in the GET request but the same happens with a POST..

Any idea?

Thanks!

Cheers.
Matias