Getting Problem with Logstash to Index Mysql Data

Elastic Stack Logstash
1

Hi All,
Details are as below. I am using Ubuntu 17.10 and installed Elastic Seach , Kibana,Logstash [Latest version 6+].
All 3 services are running perfectly.

In my project , For Testing I am trying to Input Mysql table data to Elasticserseach By Using Logstash.

Below is my conf file.

input {
jdbc {
jdbc_connection_string => "jdbc:mysql://localhost:3306/employees"
jdbc_user => "root"
jdbc_password => "nandan"
jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
statement => "select * from employees"
}
}
output {
elasticsearch {
index => "office"
document_type => "emp"
document_id => "%{ID}"
hosts => "localhost"
}
}

Conf file location
Filename is db.conf and keep it in location as /usr/share/logstash
and mysql-connector-java-5.1.46.jar file is also in the location of /usr/share/logstash/mysql-connector-java-5.1.46.jar

Executing and getting below error

Logstash
Logstash1290×127 27.9 KB

PLease let me know , how to resolve this issue.
Thanks

2

It looks like you've been following outdated documentation. Remove the "agent" argument. When in doubt look at the Logstash documentation for your version of Logstash.

3

Thanks.. I got it and now resolve other issue and Got data indexed into Elastic search.

But currently this will be like Offline Data where I have to run my Logstash file again and again.
I want to make sure when ever data will insert into Mysql, at same time data should be index into Elasticsearch, how can i achieve this ?
Thanks in advance.

4

Have a look at scheduling and state. It won't be inserted "at the same time", but with minimal delay.

5

Hi Jenni,
In case of scheduling, I can't able to achieve real time indexing and second thing is in case of scheduling , I have to run my logstash conf life in cron mode like after every few seconds.
In case Low data write this will work kind of good but scheduling will be not a good choice for REAL TIME APPLICATION.
Thanks for your reply, by this I got more information.

6

In case of scheduling, I can't able to achieve real time indexing and second thing is in case of scheduling , I have to run my logstash conf life in cron mode like after every few seconds.

The jdbc input has a built-in schedule option.

In case Low data write this will work kind of good but scheduling will be not a good choice for REAL TIME APPLICATION.

Then you'll have to look into if you can use a MySQL trigger to ping Logstash when there's a change in the table. I don't know enough about MySQL triggers to give any advice.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.