Hi,
@Tim: Was not using the x-pack-transport client. I was using the org.elasticsearch.client:transport client. Was of the impression that x-pack-transport client is only required when security feature is being used. Since I had disabled the security feature, was using the normal transport client. The code is working perfectly after using the x-pack-transport client. Perhaps the same could be mentioned in the documentation that if you are using x-pack then always x-pack-transport client should be used. (Note this was not the case in es-5-2-2, normal transport client used to work with x-pack)
@dkyle: That explains why normal transport client used to work earlier and doesn't work now. FYI: X-pack was installed in all the machines. It seems just the normal transport client cannot be used now if you are using x-pack .
Thanks all!