# Data mixed between indices

**URL:** https://discuss.elastic.co/t/data-mixed-between-indices/41355
**Category:** Logstash
**Created:** [February 10, 2016, 7:38am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355 "2016-02-10T07:38:56Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 10, 2016, 7:38am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/1 "2016-02-10T07:38:56Z")

</div>

I have a pretty simple setup. ES 2.2 and logstash 2.2.0.  
I have several conf for logstash, each one defines a port to listen to, labels data with "type=\>xyz" and puts in an index with the same name and date time reference 'index =\> "xyz-%{+YYYY.MM.dd}"'.  
Index are created with no problems. The real problem is that I find data with a type=xyz in an index named "abc" that was meant for data of type abc that comes on a specific port.  
So it was meant to be:  
port 1234 --\> type=abc --\> index=abc-yyyy.mm.ss  
port 1235 --\> type=xyz --\> index=xyz-yyyy.mm.ss  
I search the data using kibana and indexes are created using abc-\* and xyz-\*, but using old syntax yyymmdd is the same. When I explode found records I found \_index:abc and type:xyz which is impossible in my idea.  
When I have to find some data I must try on each index in kibana and I found it in any index...  
What am I doing wrong?  
Some files are as simple as being collectd five-liners...  
 ![](https://us1.discourse-cdn.com/elastic/original/2X/f/f132742c53988203de8f853464eee3b89cb954e7.JPG)  
Here is data from syslog collector (listening on udp 514) that went into the collectd index (test-cd) listening on 25826.  
Thanks

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 11, 2016, 6:00am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/2 "2016-02-11T06:00:20Z")

</div>

What does you Logstash config look like?

---

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 11, 2016, 7:10am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/3 "2016-02-11T07:10:59Z")

</div>

This is the syslog one

```auto
input {
 syslog {
  type => syslog
 }
}

output {
 elasticsearch {
  hosts => localhost
  index => "syslog-%{+YYYY.MM.dd}"
 }

 # stdout { codec => rubydebug }
}

```

This is the collectd one

```auto
input {
 udp {
port => 25826
buffer_size => 1452
codec => collectd {}
type =>collectd
 }
}

output {
        elasticsearch {
                hosts => localhost
                action => "index"
                index => "test-cd"

        }
        # stdout { codec => rubydebug }
}

```

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 11, 2016, 7:13am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/4 "2016-02-11T07:13:52Z")

</div>

If you have multiple outputs and do not use [conditionals](https://www.elastic.co/guide/en/logstash/2.2/event-dependent-configuration.html) in your configuration, Logstash will send each event to all outputs.

---

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 11, 2016, 7:33am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/5 "2016-02-11T07:33:53Z")

</div>

Maybe I'm missing something but the doc you refer explains conditionals as "if then else" things.  
There are no conditionals in my config files since there is a single output for each one.  
The formatting was not clear, but these are two SEPARATE files, one for syslog and one for collectd, each started in the conf.d directory using the daemon.  
The only double output I see is the (commented) console rubydebug line.

---

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 11, 2016, 7:38am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/6 "2016-02-11T07:38:54Z")

</div>

Ok, maybe I got something more.  
Are you saying that if I put 10 .conf files in the conf.d directory they are considered as a big one? In that case I need to put a conditional in front of each output like  
if type1 then output 1  
if type2 then output 2  
I thought that each file lived in its own "domain". I saw several examples of different files run by the daemon and noone noted cross inputs.

---

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 11, 2016, 7:42am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/7 "2016-02-11T07:42:03Z")

</div>

The syslog file above should be written as

```auto
output {

if [type]=="syslog" {
 elasticsearch {
  hosts => localhost
  index => "syslog-%{+YYYY.MM.dd}"
 }
}

```

So that if it is run as a single file it behaves normally, and when run with other files it will output only "its own" events. Is this right?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [February 11, 2016, 8:00am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/8 "2016-02-11T08:00:22Z")

</div>

Yes, that is correct. Logstash will read all config files in the directory and basically concatenate them, which is why conditionals in output and filter blocks are essential when modularising configuration.

---

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 11, 2016, 8:02am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/9 "2016-02-11T08:02:44Z")

</div>

Thank you, I totally misunderstood the meaning of having different files. I'm rewriting my conditions right now.

---

<div class="post-metadata">

### Author: ![LoZio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lozio/32/25946_2.png) [@LoZio](https://discuss.elastic.co/u/LoZio)
#### Post date: [February 11, 2016, 8:13am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/10 "2016-02-11T08:13:31Z")

</div>

And of course it is working as expected now. Thank you.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:12am UTC](https://discuss.elastic.co/t/data-mixed-between-indices/41355/11 "2017-07-06T05:12:06Z")

</div>


