Hi all,
So yesterday we installed logstash and Elasticsearch via yum from repo:
name=Elasticsearch repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/oss-7.x
(forgive me if a misuse terms, I am a software developer, not a systems engineer) on a linux server.
Installed versions:
Installed Packages
Name : elasticsearch-oss
Arch : x86_64
Version : 7.10.2
Release : 1
Size : 401 M
Repo : installed
From repo : elasticsearch-7.x
Summary : Distributed RESTful search engine built for the cloud
URL : https://www.elastic.co/
License : ASL 2.0
Installed Packages
Name : logstash-oss
Arch : x86_64
Epoch : 1
Version : 7.17.2
Release : 1
Size : 596 M
Repo : installed
From repo : elasticsearch-7.x
Summary : An extensible logging pipeline
URL : https://www.elastic.co/logstash
License : ASL 2.0
Elastic is up and running:
curl "localhost:9200"
{
"name" : "wntstfo03.izaaksuite.nl",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "wRQY0SfoTCuo5MtnxtJ_FA",
"version" : {
"number" : "7.10.2",
"build_flavor" : "oss",
"build_type" : "rpm",
"build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
"build_date" : "2021-01-13T00:42:12.435326Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
But when starting logstash with following configuration:
input {
jdbc {
jdbc_connection_string => "jdbc:oracle:thin:@wntstdb03.izaaksuite.nl:1521:wntstf2"
jdbc_user => "webnext_zaken"
jdbc_password => "webnext_zaken"
jdbc_driver_library => ""
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
statement_filepath =>"/appl/sw/webnext/logstash/config_documenten/queries/documenten.sql"
last_run_metadata_path => "/appl/sw/webnext/logstash/config_documenten/parameters/.jdbc_last_run_doc"
}
}
# The filter part of this file is commented out to indicate that it is
# optional.
#filter {
#
#}
output {
stdout { codec => rubydebug }
file {
path => ["/appl/sw/webnext/logstash/config_documenten/output/documenten.txt"]
}
elasticsearch {
hosts => ["localhost:9200"]
index => "documenten"
document_id => "%{zaakdoc_id}"
}
}
we get an error:
/usr/share/logstash/bin/logstash "--path.settings" "/appl/sw/webnext/logstash/config_documenten"
Using JAVA_HOME defined java: /usr/lib/jvm/java
WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2022-04-05T11:25:51,834][INFO ][logstash.runner ] Log4j configuration path used is: /appl/sw/webnext/logstash/config_documenten/log4j2.properties
[2022-04-05T11:25:51,892][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +indy +jit [linux-x86_64]"}
[2022-04-05T11:25:51,895][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2022-04-05T11:25:53,857][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-04-05T11:25:55,938][INFO ][org.reflections.Reflections] Reflections took 105 ms to scan 1 urls, producing 119 keys and 419 values
[2022-04-05T11:25:59,827][INFO ][logstash.outputs.elasticsearch][documenten] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2022-04-05T11:26:00,360][INFO ][logstash.outputs.elasticsearch][documenten] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2022-04-05T11:26:01,830][ERROR][logstash.javapipeline ][documenten] Pipeline error {:pipeline_id=>"documenten", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:245:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:238:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:370:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:87:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:81:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:358:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.4.1-java/lib/logstash/outputs/elasticsearch.rb:279:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/appl/sw/webnext/logstash/config_documenten/documenten.conf"], :thread=>"#<Thread:0x6e88b789 run>"}
[2022-04-05T11:26:01,834][INFO ][logstash.javapipeline ][documenten] Pipeline terminated {"pipeline.id"=>"documenten"}
[2022-04-05T11:26:01,860][ERROR][logstash.agent ] Failed to execute action {:id=>:documenten, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<documenten>, action_result: false", :backtrace=>nil}
[2022-04-05T11:26:02,000][INFO ][logstash.runner ] Logstash shut down.
[2022-04-05T11:26:02,015][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
Any help is greatly appreciated!