First, apologies if the category is incorrect, I think this question is half ES and half Kibana.
I have 10+ machines (on the same subnet) each running their own elasticsearch instance (plus logstash & logstash-forwarder). On each of these 10+ servers I'm using logstash to ingest HTTP proxy logs (local, on each of the 10+ machines). On another subnet I have a single ELK server that is receiving logs from those 10+ machines (mentioned above) via logstash-forwarder. The logs i'm sending via logstash-forwarder are not the HTTP proxy logs. I don't want to send the proxy logs via logstash-forwarder due to bandwidth (proxy logs can easily reach 4GB+ in a single day).
From my single ELK server I would like to be able to query the elasticsearch instances on the 10+ machines using Kibana. Even though the ELK server is on a different subnet I can still reach the 10+ machines running elasticsearch/logstash/logstash-forwarder. My HTTP proxy logs contain user-agent, URL, IP etc.. all in json format, I would like to be able to run a search from the ELK server (Kibana) that would run on 10+ machines to look for say a specific "user-agent"
TL;DR -- I want to search for data across 10+ independent elasticsearch instances/nodes from a single Kibana interface. What is the best way to accomplish this?
Hopefully this makes sense!