Logstash not work(

Elastic Stack Logstash
1

[2019-11-20T09:31:30,990][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.4"}
[2019-11-20T09:31:37,888][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-11-20T09:31:38,480][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-11-20T09:31:38,767][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-11-20T09:31:38,853][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-11-20T09:31:38,858][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2019-11-20T09:31:38,896][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-11-20T09:31:38,907][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-11-20T09:31:38,940][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-11-20T09:31:39,089][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xf892ed4 run>"}
[2019-11-20T09:31:39,154][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-11-20T09:31:39,576][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

2

Did you have a question?

3

I have error
this is my config

input {
jdbc {
jdbc_connection_string => "jdbc:mariadb://localhost:3306/testdb"
jdbc_user => "root"
jdbc_password => "UHzip7ugwTzocLy"
jdbc_driver_library => "/opt/mariadb-java-client-2.5.1-javadoc.jar"
jdbc_driver_class => "org.mariadb.jdbc.Driver"
statement => "SELECT * FROM testtable"
schedule => "0 * * * *"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "test-migrate"
"document_type" => "data"
}
}

this config is autoloading then system start

this is logs

[2019-11-20T16:45:57,831][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-11-20T16:45:57,861][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-11-20T16:45:57,973][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5771fb13 run>"}
[2019-11-20T16:45:58,043][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-11-20T16:45:58,407][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

not create index in elastic search

how to set up elastic with in jbdc mariadb driver auto update sql data?
very big thanks!!!!

4

It is not logging any errors, which suggests it is connecting to the database OK. Your schedule means the query will execute at the start of every hour (minute zero). Are you waiting long enough for that to happen?

5

Today, every hour, check.

No data insered in elasticsearch(

near result from command line

curl -XPOST 'http://localhost:9200/test-migrate/_search?pret ty=true' -H "Content-Type: application/json" -d '{}'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test-migrate",
"index_uuid" : "na",
"index" : "test-migrate"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test-migrate",
"index_uuid" : "na",
"index" : "test-migrate"
},
"status" : 404
}

6

The only explanation I can think of is that there is no data in testtable, so the SELECT does not return any rows, so no events are created. Does the stdout output show anything?

7

Database changed
MariaDB [testdb]> select * from testtable
-> ;
+----------+----------+-----------+-----------+----------------------------+
| PersonID | LastName | FirstName | City | Date |
+----------+----------+-----------+-----------+----------------------------+
| 4005 | Kallis | Jaques | Cape Town | 2016-05-23 16:12:03.568810 |
| 4004 | Baron | Richard | Cape Town | 2016-05-23 16:12:04.370460 |
| 4003 | McWell | Sharon | Cape Town | 2016-05-23 16:12:06.874801 |
+----------+----------+-----------+-----------+----------------------------+
3 rows in set (0.00 sec)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.