Hi,
I created two conf files one to read data from db and store in index(sysjob-index) and another to read data from log file and store in index(jupiterindex3) and i ran both conf file
bin>logstash -f logstash*.conf.
Problem:We can see db data in jupiterindex3 and log file data in sysjob-index1 in kibana which should be done ideally.
Please let us what is solution for this.
logstashSQL.conf:(read data from DB)
input { jdbc {
jdbc_driver_library => "C:\logstash-5.5.2\lib\sql\sqljdbc4-4.0.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://10.1.4.8:1111;DatabaseName=test1;"
jdbc_validate_connection => true
jdbc_user => "***"
jdbc_password => "**"
statement => "SELECT * FROM sysjobhistory where instance_id > :sql_last_value"
jdbc_paging_enabled => "true"
jdbc_page_size => "50000"
schedule => " * * * *"
use_column_value => true
tracking_column => "instance_id"
tracking_column_type => "numeric"
clean_run => true
last_run_metadata_path => "C:\logstash-5.5.2\data.logstash_jdbc_last_run"
}}
output { elasticsearch { hosts => ["localhost:9200"]
index => "sysjob-index1"
user => "elastic"
password => "test1"
}}
logstash.conf(read data from log file):
input { beats {
port => 5044
}}
filter { grok {
match => { "message" => "(?%{YEAR}-%{MONTHNUM2}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:loglevel} - 'ApplicationName':%{DATA:ApplicationName}, 'EventStatus':%{DATA:EventStatus}, 'SeverityLevel':%{DATA:SeverityLevel}, 'EventTime':(?%{YEAR}-%{MONTHNUM2}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}), 'ErrorDescription':%{DATA:ErrorDescription} "
}}}
output {elasticsearch {
hosts => ["localhost:9200"]
index => "jupiterindex3"
user => "elastic"
password => "test1"
}}
Unless you use the multi pipeline feature in Logstash 6 there is a single event pipeline. All events from all inputs will reach all outputs. If you don't want that you need to add conditionals.
Hi magnus,i read ur previous blog and understand what u suggest above.
my doubt is which one is correct.
1.input {
type=>beats
beats {
port => 5044
}}
or
2.input {
beats {
port => 5044
type=>beats
}}
And i am not able to get where i need to give condition weather after output{ or elastic{
2.input {
beats {
port => 5044
type=>beats
}}
This is correct.
And i am not able to get where i need to give condition weather after output{ or elastic{
After output {.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.