Logstash 2.1.1 - weird results with multiple conf files on WIndows

Elastic Stack Logstash
1

I am running logstash 2.1.1 on a local WIndows system with 2 conf files - one with a jdbc input that defines a custom index in the elasticsearch output, and one with a file input from a log that uses the default index (logstash-DATE). The jdbc input is reading from an Oracle table with 20 rows. I am seeing some really weird results when I run logstash against these 2 conf files simultaneously:

  1. All 20 row/documents from the db are being written to stdout twice
  2. When I do a _search all against elasticsearch, all the db documents are showing up twice, once in the default index, and once in the index defined in my jdbc conf.

It's like the conf with the file input is ignoring the input section totally, and copying the created documents from the jdbc conf into the index defined in the file conf. Below are the confs:

file.conf

The # character at the beginning of a line indicates a comment. Use

comments to describe your configuration.

input {
file {
path => "c:\openstream\ICS."
exclude => [".gz", ".txt", ".pdf"]
start_position => beginning
# codec => multiline {
# pattern => "Exception:"
# negate => true
# what => "previous"
# }
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {
grok {
match => { "message" => "%{EXCEPTION:exception}" }
patterns_dir => ["c:\logstash-2.1.1\patterns"]
}
}
output {
stdout {codec => json_lines}
elasticsearch {
hosts => "localhost:9200"
document_type => "os_log"
}
}

jdbc.conf

The # character at the beginning of a line indicates a comment. Use

comments to describe your configuration.

input {
jdbc {
jdbc_connection_string => "jdbc:oracle:thin:@//10.116.1.46:1521/bmsrpt"
jdbc_user => "bms_owner"
jdbc_password => "bms"
jdbc_validate_connection => true
jdbc_driver_library => "ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement => "SELECT * from SESSIONHISTORY"
# schedule => "*/5 * * * *"
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {

}

output {
stdout {codec => json_lines}
elasticsearch {
hosts => "localhost:9200"
index => "shistory"
document_type => "session_history"
}
}

2

Update: when I changed the backslashes to forward slashes, I then saw documents being created from the file.conf. However, both sets of documents seem to be conflated among the two indicies; i.e. it appears every document is being created in each index, instead of just the log lines going to the default index and just the db lines going to the db index.

3

When you start LS with multiple config files it merges them, so you get both stdout.
You need to use conditionals to only have each input allocated to their respective outputs.

4

how do we use conditionals to only have each input allocated to their respective outputs?

5

Please start a new thread for your question.