Why does Logstash write into two indices? (default and custom ones)

[Mr.King]

Hello,

I am using Elasticsearch 5.2 and Logstash 5.2 .

My problem is that Logstash is writing into the default index (format logstash-2017.02.15) and into my custom one. I just want it to write only into my "logstash-secure" index. How can I do?

Here is my simple Logstash configuration:

output {
    elasticsearch {
        hosts => [ "elasticsearch:9200" ]
        index => "logstash-secure"
    }
    stdout { codec => rubydebug }
}

PS: if you tell me that I have to use a custom template, could you please explain me why?

[magnusbaeck (Magnus Bäck)]

Logstash sends events to logstash-2017.02.15 because you've told it to. I suspect you have more than one configuration file in /etc/logstash/conf.d. Remember that Logstash reads all files in that directory.

[Mr.King]

Thank you Magnus for your answer.
You are right, I have an other configuration. I did not paste it because I thought there was no incidence.

Here is my first conf file:

input {
    beats {
        port => "5044"
    }
}

filter {
[some conf]
}

output {
    elasticsearch {
        hosts => [ "elasticsearch:9200" ]
    }
}

And my second conf file:

input {
  file {
    path => "/var/log/secure"
    start_position => "beginning"
  }
}
filter {
[some conf]
}
output {
    elasticsearch {
        hosts => [ "elasticsearch:9200" ]
        index => "logstash-secure"
    }
    stdout { codec => rubydebug }
}

If I understand your answer, no matter in which file the configuration is written the ouputs sections are "merged"?
If it is the case, my question would become "how could I route two inputs into two different indices"?

[magnusbaeck (Magnus Bäck)]

If I understand your answer, no matter in which file the configuration is written the ouputs sections are "merged"?

Yes.

If it is the case, my question would become "how could I route two inputs into two different indices"?

Use conditionals, e.g. based on the message type or some other field.

Accessing event data and fields | Logstash Reference [8.11] | Elastic

[Mr.King]

Thank you very much for your help Magnus.
I solved my issue with the conditional statements.
I post it for people who can wonder the same question

Here is my new first conf file:

input {
    beats {
        port => "5044"
        add_field => { "log_type" => "apache" }
    }
}
filter {
[some conf]
}
output {
    if [log_type] == "apache" {
        elasticsearch {
            hosts => [ "elasticsearch:9200" ]
        }
    }
}

And my second conf file:

input {
  file {
    path => "/var/log/secure"
    start_position => "beginning"
    add_field => { "log_type" => "secure.log" }
  }
}
filter {
[some conf]
}
output {
        if [log_type] == "secure.log" {
           elasticsearch {
               hosts => [ "elasticsearch:9200" ]
               index => "logstash-secure"
           }
       }
}