Unable to do SaveToES by user without manage privs

[Min_Qiu]

We are using org.elasticsearch:elasticsearch-spark-20_2.11:5.3.0 for EsSpark.

When we do SaveToES or SaveJsonToES, the same es user which works for logstash and curl always get denied with the following error:
action [indices:admin/refresh] is unauthorized for user [reader_writer]

This user has the role with following privs:
indices is ‘*’ with privs of write, create_index, monitor.
That is enough for logstash, why not for Spark saveToES?

When we add "manage" to the role, it is able to do SaveToES,

Does anybody have the same problem and share please?

Thanks

[james.baiera]

This is because at the end of every task, the connector attempts to refresh the write index so that the newly bulk indexed data is immediately available after the job completes. If you would like to disable this functionality, you can set es.batch.write.refresh to false in the connector's properties. Just make sure to call refresh after the job is complete or else the written data will only be available after the cluster automatically performs its own refresh. If the data does not need to be immediately visible once the job completes, then you can ignore doing the refresh and allow the cluster to refresh on its own.