Problem with downloading scan results by iteration


(abshkd) #1

I am trying to download a large result of a frequently updating ES index.

I am using python requests library to connect to the remote servers. After
starting the scrolling process and making my first 2 scroll requests I get

socket.error: [Errno 10054] An existing connection was forcibly closed by
the remote host

is this something to do with the web server of ES?

I am already hunting for python related issue but so far I have not found
any specific issue. I have tried to set the keep-alive to on/off as well as
reuse the connection pool (maybe ES doesnt like too many connections)

any clues?

thanks


(abshkd) #2

I traced the problem to an issue that I thought was solved.

When I start using the scroll_id the first two requests are fine (start the
scan request and then submit scroll_id twice) but after that the _scroll_id
seems to not change could that be a problem?


(system) #3