Basic Question on splitting data sources between 2 or more ES systems

Elastic Stack Elasticsearch
1

Its probably down to not using the correct terminology when searching this
group but whats the recommended way to handle the situation where I need
the same data source to be in 2 different elastic searches?

ES1 data sources
syslogs
firewall logs
webserver logs

ES2 data sources
Twitter
flat file
webserver logs (same as ES1)

The idea would be that each elastic search would have its own kibana but 1
would be used by sys admins and the other would be used for more business
analysis purposes

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/05a3c8d8-d1b1-4bea-8ebf-f70db8010833%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Sounds reasonable. Yeah at the moment, Kibana pointing to a single cluster
will share all the dashboards among everybody who have access to it. If you
don't want the dashboard sharing, you need 2 separate ES clusters for now,
each with its own Kibana.

However, there are some ideas in this post that you might be able to use to
configure a proxy behind 2 Kibanas (for instance) pointing to a single ES
cluster:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for that but it wasn’t so much the kibana side of things I was wondering about as I would expect to have to use separate kibanas anyway. Its more what’s the best way to set things up so that I can have the same data source in 2 different ES clusters so that its available to both but each cluster doesn’t have the extra sources required by the other?

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Binh Ly
Sent: 12 March 2014 13:56
To: elasticsearch@googlegroups.com
Subject: Re: Basic Question on splitting data sources between 2 or more ES systems

Sounds reasonable. Yeah at the moment, Kibana pointing to a single cluster will share all the dashboards among everybody who have access to it. If you don't want the dashboard sharing, you need 2 separate ES clusters for now, each with its own Kibana.

However, there are some ideas in this post that you might be able to use to configure a proxy behind 2 Kibanas (for instance) pointing to a single ES cluster:

http://www.elasticsearch.org/blog/restricting-users-kibana-filtered-aliases/

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B105%40ExMbx1.ul.campus.
For more options, visit https://groups.google.com/d/optout.

4

Oh sorry, if your feeder is Logstash, I'd probably just add to the config
output section to feed both ES clusters. :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Could that lead to delays in 1 cluster receiving data though?

From: elasticsearch@googlegroups.com [mailto:elasticsearch@googlegroups.com] On Behalf Of Binh Ly
Sent: 12 March 2014 14:15
To: elasticsearch@googlegroups.com
Subject: Re: Basic Question on splitting data sources between 2 or more ES systems

Oh sorry, if your feeder is Logstash, I'd probably just add to the config output section to feed both ES clusters. :slight_smile:

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.commailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B132%40ExMbx1.ul.campus.
For more options, visit https://groups.google.com/d/optout.

6

Yes it could - although test it to see if it is acceptable to you. If it
becomes a problem, then you can always run multiple LS feeders one per ES
cluster and then just separate the config outputs individually.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a5995252-e0db-4bbd-8768-3ce1238d9eef%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Forgive me but when you say feeders do you mean the LS actually processing
the log? Can you run multiple LS's on the same log without having them trip
over each other or end up with missing data read by the other LS first?

On Wednesday, March 12, 2014 3:12:04 PM UTC, Binh Ly wrote:

Yes it could - although test it to see if it is acceptable to you. If it
becomes a problem, then you can always run multiple LS feeders one per ES
cluster and then just separate the config outputs individually.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/40b5a5f5-42b6-4fa1-baa4-5d27639b4563%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

You can run different instances of LS each with its own config file. When
you define your file input, just point it to a unique since_db location
(that's different for each instance)

http://logstash.net/docs/1.3.3/inputs/file#sincedb_path

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5229af0a-cfbd-490d-a703-84b71e7e9d5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.