We are due to deploy Elasticsearch (Along with Graylog) to our Office in London , New York, Seattle & San Francisco
We have a a server in each remote office (Graylog & Elasticsearch). However i have a small question / Problem.
We will be logging quite a lot of information. What i would like to do is:
Main Office : London
Store all information in the cluster in Elasticsearch DB here
New York (Main US Office):
Store all Northern American office Logs in Elasticsearch Database here (NY, Seattle & San Fran logs will be saved in the Elasticsearch DB
(Seattle, Sanfran will store logs ONLY from their office. So Network devices , Syslog messages in their respective office will be stored on their own Elasticsearch database only)
I dont want Seattle & San Fran to hold the whole Elasticsearch DB as the offices are only small and only about 10 devices will be logging to ES / Graylog . Whereas NYC and UK will have > 100 devices/Servers logging to itself..
I want to be able to have UK and NYC holding ALL the information in the ES cluster and Sea and San fran to only hold their own logs but also send it to NYC so we have a backup. Is that possible?
I've seen Shard Allocation filtering but unsure on how to go ahead with it. Sorry if this is confusing it's been a project for > 6 months and ideally want to roll it out in the next 30-40 days.
Thank you for all your help,
Junior Sys Admin