Is there a request timeout parameter for spark connector?

Elastic Stack Elasticsearch
1

Occasionally I see request timeout at phase/query+fetcher/scroll. I think it's because the cluster is busy and the request didn't come back in 5 minutes (the default scroll timeout) and I see in log that the request was under path: /_search/scroll, params: {scroll=5m}.

To verify this I want to increase the request timeout, but I don't find a parameter in the spark support page to specify scroll timeout. There's a es.http.timeout but that value is default to 1m.

Is that the parameter I should use or is there another one?

Thank you very much!

2

I would set es.http.timeout to address the client side timeout for waiting on a response, and would also increase es.scroll.keepalive to increase the amount of time that the scroll is kept available.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.