I am using elasticsearch with logstash [Version : 7.9.0] to visualize a data set with kibana.
I am doing this task on Windows10.
Logstash config file -
input {
jdbc {
jdbc_driver_library => "$PATH\mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/world?autoReconnect=true&useSSL=false"
jdbc_user => root
jdbc_password => herculus
parameters => { "names" => "Kabul" }
statement => "SELECT count(*) FROM world.city where name = :names;"
schedule => " * * * * * *"
}
}
output {
elasticsearch {
document_id => "%{id}"
document_type => city
index => cities
hosts => ["http://localhost:9200"]
}
stdout{
codec => rubydebug
}
}
Response to curl command,
logstash-7.9.0\bin>curl -X GET "localhost:9200/cities/_search"
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [cities]","resource.type":"index_or_alias","resource.id":"cities","index_uuid":"_na_","index":"cities"}],"type":"index_not_found_exception","reason":"no such index [cities]","resource.type":"index_or_alias","resource.id":"cities","index_uuid":"_na_","index":"cities"},"status":404}
Also, for Get request to URL : http://localhost:9200/_cat/indices?v&pretty
O/P -
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
When I updated the logstash.conf to ,
input {
jdbc {
jdbc_driver_library => "$PATH\mysql-connector-java-5.1.46.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://localhost:3306/world?autoReconnect=true&useSSL=false"
jdbc_user => root
jdbc_password => herculus
parameters => { "names" => "Kabul" }
statement => "SELECT * FROM world.city where name = :names;"
schedule => " * * * * * *"
}
}
output {
if "world.city" in [tags] {
elasticsearch {
# document_id => "%{id}"
# document_type => city
index => "cities"
hosts => ["http://localhost:9200"]
}
}
stdout{
codec => rubydebug
}
}
I am getting response
[2020-08-26T10:28:01,229][INFO ][logstash.inputs.jdbc ][main][3b01a5ef336ad85c3946fe6866ca42078069f817e05ec92d17e8ab77baa85416] (0.002075s) SELECT * FROM world.city where name = 'Kabul';
{
"population" => 100000,
"@version" => "1",
"@timestamp" => 2020-08-26T04:58:01.231Z,
"name" => "Kabul",
"countrycode" => "IDN",
"district" => "Yasjdasjk",
"id" => 4081
}
{
"population" => 10000,
"@version" => "1",
"@timestamp" => 2020-08-26T04:58:01.231Z,
"name" => "Kabul",
"countrycode" => "IDN",
"district" => "Uhjhgjh",
"id" => 4082
}