Slowness in elasticsearch queries can be caused by any number of issues. Are you monitoring your stack? If not, I recommend installing the marvel plugin, which may give you valuable insights into where the bottleneck is.
Take a look at Kibana 4 Running ordinary search so so slow