Architecture question re. routing and multi DC

Elastic Stack Elasticsearch
1

For redundancy purposes, our system is split into two datacenters. One of
the DCs is considered central where all the backoffice systems reside and
the other is edge. Recently we started using Logstash with ElasticSearch
and Kibana. The architecture we had is:

  • Scribe server on each instance in our cluster forwards logs to a main
    scribe instance in the DC.
  • If the DC is the edge, its main scribe instance forwards all logs to
    the main scribe instance in central.
  • From the main (central) scribe server we forward message to Logstash,
    which in turn get written to ES.

Because most logs are only stored but never retrieved, to reduce the
traffic between DCs, we thought of using custom routing:

  • Have elastic search node in each DC (currently we have only one).
  • Tag each log message with the DC it's originated from and route the
    log messages according to this tag, so each DC's log messages end up in its
    own ES instance.

Will this work? Is this proper use of ElasticSearch's routing?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0017a4a8-80ca-4fcb-97df-032f9d6858c9%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

"From the main (central) scribe server we forward message to Logstash,
which in turn get written to ES."

Hey.
I wanted to know how did you infact forward the message from scribe server
to logstash? Did you do it by creating a custom logstash input plugin by
yourself?

Thanks in advance.
Regards,
Ananya

On Thursday, January 9, 2014 4:38:47 PM UTC+5:30, Arik Fraimovich wrote:

For redundancy purposes, our system is split into two datacenters. One of
the DCs is considered central where all the backoffice systems reside and
the other is edge. Recently we started using Logstash with Elasticsearch
and Kibana. The architecture we had is:

  • Scribe server on each instance in our cluster forwards logs to a
    main scribe instance in the DC.
  • If the DC is the edge, its main scribe instance forwards all logs to
    the main scribe instance in central.
  • From the main (central) scribe server we forward message to
    Logstash, which in turn get written to ES.

Because most logs are only stored but never retrieved, to reduce the
traffic between DCs, we thought of using custom routing:

  • Have Elasticsearch node in each DC (currently we have only one).
  • Tag each log message with the DC it's originated from and route the
    log messages according to this tag, so each DC's log messages end up in its
    own ES instance.

Will this work? Is this proper use of Elasticsearch's routing?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/832ee819-de12-437c-b48a-6688330c2625%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

From the main (central) scribe server we forward message to Logstash, which in turn get written to ES."

Hey.
I wanted to know how did you infact forward the message from scribe server to logstash? Did you do it by creating a custom logstash input plugin by yourself?

Thanks in advance.
Regards,
Ananya