Connection timeout when resthighlevel client is called from spark mapPartitions method

[vinayak_j]

Didnt used curl commands to test ; but have checked like if we do connection outside mapPartitions() and put sample request we get a response. If we do the same inside mapPartitions() it gives connection time out.

We use dataproc to submit spark ; since code in the driver gets response; when we do the same inside mapPartitions() it has to yield response.

Also we have referred this:

Serializing Elasticsearch clients for use when looping through Spark RDDs

Thanks! @james.baiera and @hokiegeek2 My apologies for cross posting. Please find below the current code that we have. I will try your suggestion for rewriting the code to use the rest client instead. Apologies in advance for a long list of exceptions. I am providing it as per @hokiegeek2's request for the current problem. The connection manager to Elasticsearch is as follows: object Connection { def conn() = { val port = 9300 val NodeName = "node1" val ESClusterName = "esclust…

Declaring RestHighLevelClient as private , would lead to serialization issue; so we declared it as transient.