Multiple logstash conf


(RAMESH KUMAR BAPANAPALLI) #1

Hi,
I created two conf files one to read data from db and store in index(sysjob-index) and another to read data from log file and store in index(jupiterindex3) and i ran both conf file
bin>logstash -f logstash*.conf.
Problem:We can see db data in jupiterindex3 and log file data in sysjob-index1 in kibana which should be done ideally.
Please let us what is solution for this.
logstashSQL.conf:(read data from DB)
input { jdbc {
jdbc_driver_library => "C:\logstash-5.5.2\lib\sql\sqljdbc4-4.0.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://10.1.4.8:1111;DatabaseName=test1;"
jdbc_validate_connection => true
jdbc_user => "***"
jdbc_password => "**"
statement => "SELECT * FROM sysjobhistory where instance_id > :sql_last_value"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
schedule => "
* * * *"
use_column_value => true
tracking_column => "instance_id"
tracking_column_type => "numeric"
clean_run => true
last_run_metadata_path => "C:\logstash-5.5.2\data.logstash_jdbc_last_run"
}}
output { elasticsearch { hosts => ["localhost:9200"]
index => "sysjob-index1"
user => "elastic"
password => "test1"
}}
logstash.conf(read data from log file):
input { beats {
port => 5044
}}
filter { grok {
match => { "message" => "(?%{YEAR}-%{MONTHNUM2}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:loglevel} - 'ApplicationName':%{DATA:ApplicationName}, 'EventStatus':%{DATA:EventStatus}, 'SeverityLevel':%{DATA:SeverityLevel}, 'EventTime':(?%{YEAR}-%{MONTHNUM2}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}), 'ErrorDescription':%{DATA:ErrorDescription} "
}}}
output {elasticsearch {
hosts => ["localhost:9200"]
index => "jupiterindex3"
user => "elastic"
password => "test1"
}}


(Magnus Bäck) #2

Unless you use the multi pipeline feature in Logstash 6 there is a single event pipeline. All events from all inputs will reach all outputs. If you don't want that you need to add conditionals.


(RAMESH KUMAR BAPANAPALLI) #3

Hi magnus,i read ur previous blog and understand what u suggest above.
my doubt is which one is correct.
1.input {
type=>beats
beats {
port => 5044
}}

or

2.input {
beats {
port => 5044
type=>beats
}}

And i am not able to get where i need to give condition weather after output{ or elastic{


(Magnus Bäck) #4

2.input {
beats {
port => 5044
type=>beats
}}

This is correct.

And i am not able to get where i need to give condition weather after output{ or elastic{

After output {.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.