MYSQL to ELASTICSEARCH multiple tables at once

I am porting data from mysql to elastisearch , I am using type => "table_type" in input and output to bifurcate into different tables ,data is going proper for each index however i am getting extra field named type with value which i have defined in type i.e. (table_1_type) for every record.

Below is sample config


   				jdbc {
                                       jdbc_driver_library => "/usr/share/java/mysql-connector-java-8.0.12.jar"
                                        #  jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
                                        jdbc_driver_class => "com.mysql.jdbc.Driver"
                                        jdbc_connection_string => "jdbc:mysql://MYSQL:3306/database"
                                        jdbc_user => "username"
                                        jdbc_password => "password"
                                        jdbc_paging_enabled => true
                                        jdbc_page_size => 200000
                                        type => "table_1_type"
                                        statement => "select columns from table_1"
                                        last_run_metadata_path => "/etc/logstash/conf.d/lastrun/.logstash_jdbc_test_last_run"


		 if [type] == "table_1_type" {

				elasticsearch {
					index => "table_1"
					document_type => "_doc"
					hosts => "localhost:9200"
					document_id => "%{unqiue_id}"

Use mutate+rename to move it to [@metadata][type], then you can still use it in a conditional but it will not get added to the document in elasticsearch.

@Badger :- Thanks for reply !It will be helpful if you can explain with example.

Use mutate like this

mutate { rename => { "type" => "[@metadata][type]" } }

Then anywhere where you test [type] change it to

if [@metadata][type] == "table_1_type" {

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.