For redundancy purposes, our system is split into two datacenters. One of
the DCs is considered central where all the backoffice systems reside and
the other is edge. Recently we started using Logstash with ElasticSearch
and Kibana. The architecture we had is:
- Scribe server on each instance in our cluster forwards logs to a main
scribe instance in the DC.
- If the DC is the edge, its main scribe instance forwards all logs to
the main scribe instance in central.
- From the main (central) scribe server we forward message to Logstash,
which in turn get written to ES.
Because most logs are only stored but never retrieved, to reduce the
traffic between DCs, we thought of using custom routing:
- Have elastic search node in each DC (currently we have only one).
- Tag each log message with the DC it's originated from and route the
log messages according to this tag, so each DC's log messages end up in its
own ES instance.
Will this work? Is this proper use of ElasticSearch's routing?
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0017a4a8-80ca-4fcb-97df-032f9d6858c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.