Hello,
I'm new to the elk stack.
Operating system: centos
ELK version 6.7
I'm trying to migrate some data from postgresql to the elastic search using logstash, but it does not work!
I have the file called: /etc/logstash/conf.d/teste.conf
with the content:
input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://10.17.0.36:5432/sagu"
jdbc_driver_library => "/etc/logstash/conf.d/postgresql-9.4.1212.jre6.jar"
jdbc_user => "****"
jdbc_password => "*****"
jdbc_driver_class => "org.postgresql.Driver"
schedule => "0 * * * *"
statement => "SELECT * FROM public.elastic_sagu_cursos"
use_column_value => true
tracking_column => "curso_id"
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
document_id => "%{curso_id}"
document_type => "teste"
index => "teste_teste"
codec => "json"
}
}
When I run the command:
./bin/logstash -f /etc/logstash/conf.d/teste.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-05-23 11:48:15.472 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-05-23 11:48:15.493 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.7.1"}
[WARN ] 2019-05-23 11:48:24.168 [Converge PipelineAction::Create<main>] elasticsearch - You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch index=>"teste_teste", codec=><LogStash::Codecs::JSON id=>"json_3d2476fd-24ce-45aa-8deb-a357bbb7a85e", enable_metric=>true, charset=>"UTF-8">, id=>"f35a1bb53db98aa3cf5c610dbf737d5af3f878ffb1079f5a78e59017d34670ab", document_id=>"%{curso_id}", hosts=>[http://localhost:9200], document_type=>"teste", enable_metric=>true, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>false, ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[INFO ] 2019-05-23 11:48:24.294 [Converge PipelineAction::Create<main>] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2019-05-23 11:48:24.884 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-05-23 11:48:25.121 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-05-23 11:48:25.340 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>6}
[WARN ] 2019-05-23 11:48:25.343 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[INFO ] 2019-05-23 11:48:25.372 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[INFO ] 2019-05-23 11:48:25.389 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[INFO ] 2019-05-23 11:48:25.442 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] 2019-05-23 11:48:25.792 [Converge PipelineAction::Create<main>] pipeline - Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x283da613 sleep>"}
[INFO ] 2019-05-23 11:48:25.889 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-05-23 11:48:26.328 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rufus-scheduler-3.0.9/lib/rufus/scheduler/cronline.rb:77: warning: constant ::Fixnum is deprecated
and data migration does not work!
I already tested this code in different environment and it works, however in this server with centos does not work: /