Logging ES queries in elasticsearch-spark-30

Elastic Stack Elasticsearch
1

Is there a way to log Elasticsearch queries issued by the elasticsearch-spark-30_2.12 library?

What I tried:

  1. Setting everything to trace log level. URLs of requests sent to Elasticsearch were printed, but not the body of the POST requests. Since all the query was in the body this hardly helps.
  2. Enabling logging as documented for Elasticsearch High-Level Java Client, i.e. setting logging.level.tracer=TRACE. That did not produce any results.
2

I haven't tried it yet, but I believe if you set org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransport logging to trace for your spark tasks it will log out the request body. elasticsearch-hadoop/CommonsHttpTransport.java at master · elastic/elasticsearch-hadoop · GitHub

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.