Multiple inputs and multiple outputs

Elastic Stack Logstash
1

Hi there,

I'm trying to set multiple inputs and multiple outputs to different elasticsearch indexes ... and something I am doing wrong because everything is going to the else.

Here is my config:

 input {
    jdbc {
        jdbc_connection_string => "jdbc:mysql:127.0.0.1:3306/whatever"
        jdbc_user => "xxx"
        jdbc_password => "yyy"
        jdbc_driver_library => "mysql-connector-java-5.1.41.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        schedule => "* * * * *"
        statement => "SELECT * from table1 WHERE updated_at > :sql_last_value order by updated_at"
        use_column_value => true
        tracking_column => updated_at
        type => "table1"
        last_run_metadata_path => "/opt/logstash-5.4.0/sql-last-values/table1"
    }
      jdbc {
        jdbc_connection_string => "jdbc:mysql:127.0.0.1:3306/whatever"
        jdbc_user => "xxx"
        jdbc_password => "yyy"
        jdbc_driver_library => "mysql-connector-java-5.1.41.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        schedule => "* * * * *"
        statement => "SELECT * from table2 WHERE updated_at > :sql_last_value order by updated_at"
        use_column_value => true
        tracking_column => updated_at
        type => "table2"
        last_run_metadata_path => "/opt/logstash-5.4.0/sql-last-values/table2"
    }
}
output {
	if [type] == "table1" {
           elasticsearch {
                hosts => ["localhost:9200"]
                index => "table1"
                document_type => "table1"
                document_id => "%{id}"
        }
        file {
                codec => json_lines
                path => "/opt/logstash-5.4.0/logs/table1.log"
        }

    } else if [type] == "table2" {
           elasticsearch {
                hosts => ["localhost:9200"]
                index => "table2"
                document_type => "table2"
                document_id => "%{id}"
        }
    } else {
    	 file {
                codec => json_lines
                path => "/opt/logstash-5.4.0/logs/unknown.log"
        	}

    }
}

What am I doing wrong ? everything is going to the else block, to the /opt/logstash-5.4.0/logs/unknown.log

thank you in advance

2

Please show an example line from unknown.log.

3

I solved !
using tags ...

input {
jdbc {
...
tags => "table1"
...
}
jdbc {
...
tags => "table2"
...
}
}
output {
if "table1" in [tags] {

}

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

5

If you don't want to have the 'type' and/or 'tags' field in the documents that are output, you can also consider adding metadata which will be used for directing data to different outputs, and removing the 'type' and/or 'tags' fields. I have written a blog post that demonstrates this approach at https://alexmarquardt.com/2018/08/31/using-logstash-to-drive-filtered-data-from-a-single-source-into-multiple-output-destinations/.