Logging in ES Hadoop Jar

(Bijuv V) #1

We are using ESHadoop2.1.0 jar inside the Hadoop environment to sync the data from the Hadoop to elastic search. We are facing some issues in the data flow and hence wanted to enable the debugging of the ES Hadoop flow. I went through the code and figured out the logger statements that need to be added.

I added the following in the Oozie log config through Ambari.

log4j.logger.org.elasticsearch.hadoop.rest=DEBUG, oozie
log4j.logger.org.elasticsearch.hadoop.hive=DEBUG, oozie
log4j.logger.org.elasticsearch.hadoop.mr=DEBUG, oozie

And I executed the hql through oozie.

The issue that I face is that the logs are not printed or printed it somewhere else.

Can you please help me with these ?

a. Are the log4j statements correct? I need to track the rest calls made to ES. Also want to know the processing that is happening.

b. Also , where do I set the log4j statements. I execute the hql through a oozie workflow.

c. Where can I see the logs printed.

(Costin Leau) #2

The docs provide a dedicated section on logging.
Since es-hadoop is a library, the logs show up on the nodes where the task is running. This is true for all jars/libraries and it is not specific for ES-Hadoop.

(system) #3