I am having permission issues when connecting to secured AWS elasticsearch cluster using the es-hadoop library. If I keep the cluster policies open to all, I am able to successfully push data into elasticsearch. When the cluster is secured with AWS IAM roles/policies I am unable to connect. I tried providing the AWS access key and secret key as the es.net.http.auth.user and es.net.http.auth.pass in the es-hadoop configuration but I still get the forbidden error. Has anybody successfully connected to a secured AWS elasticsearch cluster via es-hadoop?
AWS IAM works by adding a token/signature per request. The SSL support is not enough in this case (it secures the transport but IAM is separate from this).
The easiest way to go around this is to use an IAM proxy (since it is based on REST/HTTP) along with the proxy support in ES-Hadoop.
Is modifying the library code to use AWS signature version 4 to sign HTTP requests to elasticsearch a feasible option?
If by feasible you mean if it can be done - yes, without much effort (of course some familiarity with the both code bases is required). If however you mean making this feature part of the official product/distribution I'm afraid the answer is no.