In my local ELK server, I am searching the ELK using scrolls, the scroll Id is generated for the initial request.
However whenever requested for the next set of records, using the scroll ID generated from the first request. It is giving empty results.
The below logs are observed in elastic search logs,
[2019-04-02T09:33:52,546][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] Node [KfIfmJS1QTaZu4JjrvyZbQ] not available for scroll request [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAFixFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvRYybnNNWDRxT1JxbVdNNG83SW1lZHlBAAAAAAAAWL4WMm5zTVg0cU9ScW1XTTRvN0ltZWR5QQAAAAAAAFiwFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvxYybnNNWDRxT1JxbVdNNG83SW1lZHlB]
[2019-04-02T09:33:52,547][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] Node [2nsMX4qORqmWM4o7ImedyA] not available for scroll request [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAFixFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvRYybnNNWDRxT1JxbVdNNG83SW1lZHlBAAAAAAAAWL4WMm5zTVg0cU9ScW1XTTRvN0ltZWR5QQAAAAAAAFiwFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvxYybnNNWDRxT1JxbVdNNG83SW1lZHlB]
[2019-04-02T09:33:52,547][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] Node [2nsMX4qORqmWM4o7ImedyA] not available for scroll request [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAFixFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvRYybnNNWDRxT1JxbVdNNG83SW1lZHlBAAAAAAAAWL4WMm5zTVg0cU9ScW1XTTRvN0ltZWR5QQAAAAAAAFiwFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvxYybnNNWDRxT1JxbVdNNG83SW1lZHlB]
[2019-04-02T09:33:52,547][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] Node [KfIfmJS1QTaZu4JjrvyZbQ] not available for scroll request [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAFixFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvRYybnNNWDRxT1JxbVdNNG83SW1lZHlBAAAAAAAAWL4WMm5zTVg0cU9ScW1XTTRvN0ltZWR5QQAAAAAAAFiwFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvxYybnNNWDRxT1JxbVdNNG83SW1lZHlB]
[2019-04-02T09:33:52,547][DEBUG][o.e.a.s.TransportSearchScrollAction] [node-2] Node [2nsMX4qORqmWM4o7ImedyA] not available for scroll request [DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAFixFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvRYybnNNWDRxT1JxbVdNNG83SW1lZHlBAAAAAAAAWL4WMm5zTVg0cU9ScW1XTTRvN0ltZWR5QQAAAAAAAFiwFktmSWZtSlMxUVRhWnU0SmpydnlaYlEAAAAAAABYvxYybnNNWDRxT1JxbVdNNG83SW1lZHlB]
Earlier the scroll is working with out any issue.
I have cleared all the scrolls if any thing opened but not closed, using the API https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html#_clear_scroll_api
However the problem persists.
How to fix this issue.
Do we need to cleanup any thing from elastic side.
The number of search records to be read from elastic is around 2 million.
Setup Details: There are two clusters and each cluster having two nodes.
Out of these two clusters, one cluster is having node with better configuration in terms of RAM and number of cores. When the scroll request is sent to this cluster it is working however returned partial details. When the scroll request is sent to the other cluster it is giving the empty results straight away.
Do we need to increase any RAM for the elastic search service? It may not be the solution since earlier scroll worked perfectly using the same RAM.
Any other cleanup need to be done for elastic search?
My ELK version is 5.4.0