# Basic Question on splitting data sources between 2 or more ES systems **URL:** \ **Category:** Elasticsearch\ **Created:** [March 12, 2014, 11:12am UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316 "2014-03-12T11:12:05Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mobcdi](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@mobcdi](https://discuss.elastic.co/u/mobcdi)\ **Post date:** [March 12, 2014, 11:12am UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/1 "2014-03-12T11:12:05Z")
Its probably down to not using the correct terminology when searching this group but whats the recommended way to handle the situation where I need the same data source to be in 2 different elastic searches? _ES1 data sources_ syslogs firewall logs webserver logs _ES2 data sources_ Twitter flat file webserver logs (same as ES1) The idea would be that each elastic search would have its own kibana but 1 would be used by sys admins and the other would be used for more business analysis purposes -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/05a3c8d8-d1b1-4bea-8ebf-f70db8010833%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/05a3c8d8-d1b1-4bea-8ebf-f70db8010833%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 12, 2014, 1:56pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/2 "2014-03-12T13:56:00Z")
Sounds reasonable. Yeah at the moment, Kibana pointing to a single cluster will share all the dashboards among everybody who have access to it. If you don't want the dashboard sharing, you need 2 separate ES clusters for now, each with its own Kibana. However, there are some ideas in this post that you might be able to use to configure a proxy behind 2 Kibanas (for instance) pointing to a single ES cluster: > **[Restricting Users for Kibana with Filtered Aliases | Elastic](https://www.elastic.co/blog/restricting-users-kibana-filtered-aliases)** > > Update: After some lengthy review of the Nginx configurations we discovered that it’s not possible to lock down the aliases based on the remote user due to limitations with Nginx location expres... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mobcdi](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@mobcdi](https://discuss.elastic.co/u/mobcdi)\ **Post date:** [March 12, 2014, 2:04pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/3 "2014-03-12T14:04:23Z")
Thanks for that but it wasn’t so much the kibana side of things I was wondering about as I would expect to have to use separate kibanas anyway. Its more what’s the best way to set things up so that I can have the same data source in 2 different ES clusters so that its available to both but each cluster doesn’t have the extra sources required by the other? From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Binh Ly Sent: 12 March 2014 13:56 To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Re: Basic Question on splitting data sources between 2 or more ES systems Sounds reasonable. Yeah at the moment, Kibana pointing to a single cluster will share all the dashboards among everybody who have access to it. If you don't want the dashboard sharing, you need 2 separate ES clusters for now, each with its own Kibana. However, there are some ideas in this post that you might be able to use to configure a proxy behind 2 Kibanas (for instance) pointing to a single ES cluster: ## [http://www.elasticsearch.org/blog/restricting-users-kibana-filtered-aliases/](http://www.elasticsearch.org/blog/restricting-users-kibana-filtered-aliases/) You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe](https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe). To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a97987a9-e2cc-4b8e-857e-1871dd607816%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B105%40ExMbx1.ul.campus](https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B105%40ExMbx1.ul.campus). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 12, 2014, 2:14pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/4 "2014-03-12T14:14:36Z")
Oh sorry, if your feeder is Logstash, I'd probably just add to the config output section to feed both ES clusters. 🙂 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mobcdi](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@mobcdi](https://discuss.elastic.co/u/mobcdi)\ **Post date:** [March 12, 2014, 2:21pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/5 "2014-03-12T14:21:15Z")
Could that lead to delays in 1 cluster receiving data though? From: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) [[mailto:elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com)] On Behalf Of Binh Ly Sent: 12 March 2014 14:15 To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Re: Basic Question on splitting data sources between 2 or more ES systems ## Oh sorry, if your feeder is Logstash, I'd probably just add to the config output section to feed both ES clusters. 🙂 You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe](https://groups.google.com/d/topic/elasticsearch/50OvM5LYU2I/unsubscribe). To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com)[mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3891a9b9-cc2f-4f74-ba69-eb6d0132f370%40googlegroups.com?utm_medium=email&utm_source=footer). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B132%40ExMbx1.ul.campus](https://groups.google.com/d/msgid/elasticsearch/36667CDCAAF70140AE7738BB93CA8C9605B132%40ExMbx1.ul.campus). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 12, 2014, 3:12pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/6 "2014-03-12T15:12:04Z")
Yes it could - although test it to see if it is acceptable to you. If it becomes a problem, then you can always run multiple LS feeders one per ES cluster and then just separate the config outputs individually. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a5995252-e0db-4bbd-8768-3ce1238d9eef%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a5995252-e0db-4bbd-8768-3ce1238d9eef%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mobcdi](https://avatars.discourse-cdn.com/v4/letter/m/e480ec/32.png) [@mobcdi](https://discuss.elastic.co/u/mobcdi)\ **Post date:** [March 14, 2014, 2:38pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/7 "2014-03-14T14:38:32Z")
Forgive me but when you say feeders do you mean the LS actually processing the log? Can you run multiple LS's on the same log without having them trip over each other or end up with missing data read by the other LS first? On Wednesday, March 12, 2014 3:12:04 PM UTC, Binh Ly wrote: > Yes it could - although test it to see if it is acceptable to you. If it > becomes a problem, then you can always run multiple LS feeders one per ES > cluster and then just separate the config outputs individually. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/40b5a5f5-42b6-4fa1-baa4-5d27639b4563%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/40b5a5f5-42b6-4fa1-baa4-5d27639b4563%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 14, 2014, 7:02pm UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/8 "2014-03-14T19:02:01Z")
You can run different instances of LS each with its own config file. When you define your file input, just point it to a unique since\_db location (that's different for each instance) [http://logstash.net/docs/1.3.3/inputs/file#sincedb\_path](http://logstash.net/docs/1.3.3/inputs/file#sincedb_path) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5229af0a-cfbd-490d-a703-84b71e7e9d5a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5229af0a-cfbd-490d-a703-84b71e7e9d5a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:43am UTC](https://discuss.elastic.co/t/basic-question-on-splitting-data-sources-between-2-or-more-es-systems/16316/9 "2017-07-06T01:43:06Z")