# Mesh Configuration

**URL:** https://discuss.elastic.co/t/mesh-configuration/208609
**Category:** Elasticsearch
**Created:** [November 20, 2019, 4:51am UTC](https://discuss.elastic.co/t/mesh-configuration/208609 "2019-11-20T04:51:14Z")
**Posts on this page:** 8
**Page:** 1

<div class="post-metadata">

### Author: ![JohnM](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnm/32/49468_2.png) [@JohnM](https://discuss.elastic.co/u/JohnM)
#### Post date: [November 20, 2019, 4:51am UTC](https://discuss.elastic.co/t/mesh-configuration/208609/1 "2019-11-20T04:51:14Z")

</div>

Hi everyone,

I thought this works but it seems not..

Network devices send syslog to Logstash-1 and Logstash-2  
So Logstash-1 and Logstash-2 have same logs received. \< Confirmed.

Logstash-1 sends logs to ES-A cluster (ES-A1, ES-A2, and ES-A3)  
Logstash-2 sends logs to ES-A cluster (ES-A1, ES-A2, and ES-A3)  
ES-A cluster can get confused if 2 logstash servers send the same log to him?

Logstash-1 sends logs to ES-B cluster (ES-B1, ES-B2, and ES-B3)  
Logstash-2 sends logs to ES-B cluster (ES-B1, ES-B2, and ES-B3)  
ES-B cluster can get confused if 2 logstash servers send the same log to him?

Kibana-1 is looking at ES-A cluster  
Kibana-2 is looking at ES-B cluster

When you compare a search result of Kibana-1 and Kibana-2 it is not identical.

Is this something we shouldn't do, or this mesh config should work?

Thank you for your advice in advance.

Logstash 7.4  
ES 7.3  
Kibana 7.3

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 20, 2019, 6:31am UTC](https://discuss.elastic.co/t/mesh-configuration/208609/2 "2019-11-20T06:31:05Z")

</div>

> [@JohnM](#):
>
> ES-A cluster can get confused if 2 logstash servers send the same log to him?

If you have two Logstash instances sending the same data to Elasticsearch one of two things will happen. If you specify a document ID in your output you will get an insert and an update for each document, which is basically twice the indexing load. If you allow Elasticsearch to assign document IDs you will instead get duplicates in your index, which will affect the results in Kibana.

It is quite predictable, so I would not call it confusion.

> [@JohnM](#):
>
> When you compare a search result of Kibana-1 and Kibana-2 it is not identical.

Can you show us your Logstash config, especially the output part? How are they not identical?

---

<div class="post-metadata">

### Author: ![JohnM](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnm/32/49468_2.png) [@JohnM](https://discuss.elastic.co/u/JohnM)
#### Post date: [November 20, 2019, 6:45am UTC](https://discuss.elastic.co/t/mesh-configuration/208609/3 "2019-11-20T06:45:06Z")

</div>

Thanks for reply.  
Here is our logstash output, no have no document id specified.

```
output {
elasticsearch {
  hosts => ["es-a1.xxx:9200", "es-a2.xxx:9200", "es-a3.xxx:9200", "es-b1.xxx:9200", "es-b2.xxx:9200", "es-b3.xxx:9200"]
  manage_template => false
  index => "logstash-core-%{+YYYY.MM.dd}"
}

```

Both Logstash-1 and Logstash-2 have exact same output. We push via puppet so it can't be different.

When we get a result of kibana search, it is something like;  
Kibana-1 has 38 records  
Kibana-2 has 28 records  
about 15 records are identical but the rest of records are unique.  
Make sense?

Thanks

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 20, 2019, 6:46am UTC](https://discuss.elastic.co/t/mesh-configuration/208609/4 "2019-11-20T06:46:58Z")

</div>

That configuration will end up sending each bulk request to one of the listed nodes, so it can go either to cluster A or cluster B, but not both. If you want to send to multiple clusters you will need one elasticsearch output per cluster.

---

<div class="post-metadata">

### Author: ![JohnM](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnm/32/49468_2.png) [@JohnM](https://discuss.elastic.co/u/JohnM)
#### Post date: [November 20, 2019, 7:12am UTC](https://discuss.elastic.co/t/mesh-configuration/208609/5 "2019-11-20T07:12:31Z")

</div>

Chris, It is really helpful information.  
I had a quick search on output syntax but while you are there could you confirm this please?

This will do?

```
output {
elasticsearch {
  hosts => ["es-a1.xxx:9200", "es-a2.xxx:9200", "es-a3.xxx:9200"]
  hosts => ["es-b1.xxx:9200", "es-b2.xxx:9200", "es-b3.xxx:9200"]
  manage_template => false
  index => "logstash-core-%{+YYYY.MM.dd}"
}

```

Or we have to create 2 outputs?

```
output {
elasticsearch {
  hosts => ["es-a1.xxx:9200", "es-a2.xxx:9200", "es-a3.xxx:9200"]
  manage_template => false
  index => "logstash-core-%{+YYYY.MM.dd}"
}

```

and

```
output {
elasticsearch {
  hosts => ["es-b1.xxx:9200", "es-b2.xxx:9200", "es-b3.xxx:9200"]
  manage_template => false
  index => "logstash-core-%{+YYYY.MM.dd}"
}

```

Sorry Chris for asking too much..

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 20, 2019, 8:18am UTC](https://discuss.elastic.co/t/mesh-configuration/208609/6 "2019-11-20T08:18:38Z")

</div>

You need two separate outputs.

---

<div class="post-metadata">

### Author: ![JohnM](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/johnm/32/49468_2.png) [@JohnM](https://discuss.elastic.co/u/JohnM)
#### Post date: [November 20, 2019, 11:23pm UTC](https://discuss.elastic.co/t/mesh-configuration/208609/7 "2019-11-20T23:23:47Z")

</div>

Excellent!!  
Thanks for all your help. Problem resolved.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 18, 2019, 11:23pm UTC](https://discuss.elastic.co/t/mesh-configuration/208609/8 "2019-12-18T23:23:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
