I am still learning about elasticsearch and logstash, I am trying to index a table from postgresDB but every time I run it, it keeps shutting down
here's my logstash config:
# file: index1.conf
input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://localhost/myschema?user=postgres&password=[PWORD]"
jdbc_user => "postgres"
jdbc_driver_library => ""
jdbc_driver_class => "org.postgresql.Driver"
statement => "SELECT a.app_id, a.text, CASE WHEN b.granted = 'f' THEN 0 ELSE 1 END as isgrant FROM db1.tableA as a LEFT JOIN db2.tableA as b ON a.app_id = b.app_id LIMIT 15000000 OFFSET 0"
}
}
output {
elasticsearch {
index => "index1"
document_id => "%{app_id}"
hosts => "http://localhost:9200"
}
}
and here's the response:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2019-08-05 19:14:53.452 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2019-08-05 19:14:53.467 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.2.0"}
[INFO ] 2019-08-05 19:14:56.454 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[WARN ] 2019-08-05 19:14:56.647 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ] 2019-08-05 19:14:56.824 [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[WARN ] 2019-08-05 19:14:56.826 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2019-08-05 19:14:56.848 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[INFO ] 2019-08-05 19:14:56.904 [Ruby-0-Thread-5: :1] elasticsearch - Using default mapping template
[WARN ] 2019-08-05 19:14:56.936 [[main]-pipeline-manager] LazyDelegatingGauge - A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been create for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[INFO ] 2019-08-05 19:14:56.940 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>24, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>3000, :thread=>"#<Thread:0x2effd3df run>"}
[INFO ] 2019-08-05 19:14:57.668 [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[INFO ] 2019-08-05 19:14:57.883 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2019-08-05 19:14:57.947 [Ruby-0-Thread-1: /usr/share/logstash/lib/bootstrap/environment.rb:6] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2019-08-05 19:14:58.209 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2019-08-05 19:15:35.491 [[main]<jdbc] jdbc - (36.375952s) SELECT a.app_id, a.text as claim, CASE WHEN b.granted = 'f' THEN 0 ELSE 1 END as isgrant FROM patent_app_v2.claim as a LEFT JOIN patent_app_v2.application_update as b ON a.app_id = b.app_id LIMIT 15000000 OFFSET 0
[INFO ] 2019-08-05 19:24:45.260 [LogStash::Runner] runner - Logstash shut down.
I don't know where the problem is, any help will be appreciated.