I am running logstash 2.1.1 on a local WIndows system with 2 conf files - one with a jdbc input that defines a custom index in the elasticsearch output, and one with a file input from a log that uses the default index (logstash-DATE). The jdbc input is reading from an Oracle table with 20 rows. I am seeing some really weird results when I run logstash against these 2 conf files simultaneously:
- All 20 row/documents from the db are being written to stdout twice
- When I do a _search all against elasticsearch, all the db documents are showing up twice, once in the default index, and once in the index defined in my jdbc conf.
It's like the conf with the file input is ignoring the input section totally, and copying the created documents from the jdbc conf into the index defined in the file conf. Below are the confs:
file.conf
The # character at the beginning of a line indicates a comment. Use
comments to describe your configuration.
input {
file {
path => "c:\openstream\ICS."
exclude => [".gz", ".txt", ".pdf"]
start_position => beginning
# codec => multiline {
# pattern => "Exception:"
# negate => true
# what => "previous"
# }
}
}
The filter part of this file is commented out to indicate that it is
optional.
filter {
grok {
match => { "message" => "%{EXCEPTION:exception}" }
patterns_dir => ["c:\logstash-2.1.1\patterns"]
}
}
output {
stdout {codec => json_lines}
elasticsearch {
hosts => "localhost:9200"
document_type => "os_log"
}
}
jdbc.conf
The # character at the beginning of a line indicates a comment. Use
comments to describe your configuration.
input {
jdbc {
jdbc_connection_string => "jdbc:oracle:thin:@//10.116.1.46:1521/bmsrpt"
jdbc_user => "bms_owner"
jdbc_password => "bms"
jdbc_validate_connection => true
jdbc_driver_library => "ojdbc7.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement => "SELECT * from SESSIONHISTORY"
# schedule => "*/5 * * * *"
}
}
The filter part of this file is commented out to indicate that it is
optional.
filter {
}
output {
stdout {codec => json_lines}
elasticsearch {
hosts => "localhost:9200"
index => "shistory"
document_type => "session_history"
}
}