Is there a request timeout parameter for spark connector?

Occasionally I see request timeout at phase/query+fetcher/scroll. I think it's because the cluster is busy and the request didn't come back in 5 minutes (the default scroll timeout) and I see in log that the request was under path: /_search/scroll, params: {scroll=5m}.

To verify this I want to increase the request timeout, but I don't find a parameter in the spark support page to specify scroll timeout. There's a es.http.timeout but that value is default to 1m.

Is that the parameter I should use or is there another one?

Thank you very much!

I would set es.http.timeout to address the client side timeout for waiting on a response, and would also increase es.scroll.keepalive to increase the amount of time that the scroll is kept available.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.