Logstash not work(

jleLLlka_Kozlov · November 20, 2019, 9:48am

[2019-11-20T09:31:30,990][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.4"}
[2019-11-20T09:31:37,888][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-11-20T09:31:38,480][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://localhost:9200/]}}
[2019-11-20T09:31:38,767][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-11-20T09:31:38,853][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-11-20T09:31:38,858][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2019-11-20T09:31:38,896][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-11-20T09:31:38,907][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-11-20T09:31:38,940][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-11-20T09:31:39,089][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0xf892ed4 run>"}
[2019-11-20T09:31:39,154][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-11-20T09:31:39,576][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

Badger · November 20, 2019, 2:14pm

Did you have a question?

jleLLlka_Kozlov · November 20, 2019, 4:56pm

I have error
this is my config

input {
jdbc {
jdbc_connection_string => "jdbc:mariadb://localhost:3306/testdb"
jdbc_user => "root"
jdbc_password => "UHzip7ugwTzocLy"
jdbc_driver_library => "/opt/mariadb-java-client-2.5.1-javadoc.jar"
jdbc_driver_class => "org.mariadb.jdbc.Driver"
statement => "SELECT * FROM testtable"
schedule => "0 * * * *"
}
}
output {
stdout { codec => json_lines }
elasticsearch {
"hosts" => "localhost:9200"
"index" => "test-migrate"
"document_type" => "data"
}
}

this config is autoloading then system start

this is logs

[2019-11-20T16:45:57,831][INFO ][logstash.outputs.elasticsearch] Using default mapping template
[2019-11-20T16:45:57,861][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-11-20T16:45:57,973][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x5771fb13 run>"}
[2019-11-20T16:45:58,043][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2019-11-20T16:45:58,407][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

not create index in elastic search

how to set up elastic with in jbdc mariadb driver auto update sql data?
very big thanks!!!!

Badger · November 20, 2019, 5:58pm

It is not logging any errors, which suggests it is connecting to the database OK. Your schedule means the query will execute at the start of every hour (minute zero). Are you waiting long enough for that to happen?

jleLLlka_Kozlov · November 20, 2019, 7:31pm

Today, every hour, check.

No data insered in elasticsearch(

near result from command line

curl -XPOST 'http://localhost:9200/test-migrate/_search?pret ty=true' -H "Content-Type: application/json" -d '{}'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test-migrate",
"index_uuid" : "na",
"index" : "test-migrate"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index",
"resource.type" : "index_or_alias",
"resource.id" : "test-migrate",
"index_uuid" : "na",
"index" : "test-migrate"
},
"status" : 404
}

Badger · November 20, 2019, 7:40pm

The only explanation I can think of is that there is no data in testtable, so the SELECT does not return any rows, so no events are created. Does the stdout output show anything?

jleLLlka_Kozlov · November 20, 2019, 7:45pm

system · December 18, 2019, 7:52pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.