Data Import to Elasticsearch from MySQL

Elastic Stack Logstash
1

i use the following configuration file:

input {
    jdbc {
       
        jdbc_connection_string => "jdbc:mysql://localhost:3306/live_test"
        jdbc_user => "teddy"
		jdbc_password => "cdac"
        jdbc_driver_library => "G:\teddy\mysql-connector-java.jar"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
		statement => "SELECT a.NUM_accepted_percent,m.enm_tender_type FROM	WMS_BW_CONTRACT_AWARD a,wms_bw_tender_type_master m	WHERE	m.num_tender_type_id = a.num_agreement_type;"
		jdbc_paging_enabled=>true;
		jdbc_page_size=>500000;
    }
}

output {
	elasticsearch {
		index => "bill1"
	} 	
}

But the logstash does not send the data to elasticsearch??
Any idea how to perform the mysql query in logstash??

2

Temporarily replace the elasticsearch output with a stdout { codec => rubydebug } output to verify that the input is working as expected.

You should also inspect your Logstash log file. If it has trouble sending events to ES it'll tell you about it.

1 Like
3

It did work but everytime I push data into elasticsearch some of the records are skipped.
Eg: the above query has 357157 records but after the logstash operation only 357133 are available.

This happens everytime when the number of records are more.

If we perform joins as I have done in the query, does the performance of logstash is affected??

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.