There are several ways to run spark and I'm not sure which one you're using. But the classpath for spark-shell on the executor node is not necessarily the same as the one that's actually used by the executors. I'm not familiar with running spark in kubernetes, so I don't know what the best advice is for getting the classpath of the executors or driver. If you're able to ssh onto the nodes you might be able to get it with something like ps -ef | grep spark.