Logstash enable ssl to connect ElasticSearch but cannot run logstash and get error

chauncy1 · April 1, 2020, 11:41am

Hi，

I have configured the x_pack of ElasticSearch and SSL of ElasticSearch and Kibana.
But when I try to config logstash to connect ElasticSearch with SSL, I got some error.

Version: Logstash7.3.2, ElasticSearch7.3.2, Kibana7.3.2
ElasticSearch has 3 node, kibana and logstash both are single node.

Here is the configuration of ES in elasticsearch.yml ：

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/${node.name}.p12
xpack.security.transport.ssl.truststore.path: certs/${node.name}.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: certs/${node.name}.p12
xpack.security.http.ssl.truststore.path: certs/${node.name}.p12

Here is the configuration of Kibana in kibana.yml：

elasticsearch.username: "kibana"
elasticsearch.password: "password"

elasticsearch.hosts: ["https://53.99.41.73:9200"]
elasticsearch.ssl.certificateAuthorities: [ "/opt/es/kibana/kibana-7.3.2-linux-x86_64/certs/elastic-ca.pem" ]  
elasticsearch.ssl.verificationMode: certificate

server.ssl.enabled: true
server.ssl.certificate: /opt/es/kibana/kibana-7.3.2-linux-x86_64/certs/ca/ca.crt
server.ssl.key: /opt/es/kibana/kibana-7.3.2-linux-x86_64/certs/ca/ca.key

Until this point, ES and Kibana are running normally with SSL.

Then I config my logstash in logstash.conf:

output{
        elasticsearch{
            hosts => "53.99.41.73:9200"
            index => "monitor_message_flow"
            document_id => "%{mmfid}"
            user => "logstash_internal"
            password => "password"
            ssl => true
            cacert => "/opt/es/logstash/logstash-7.3.2/certs/elastic-ca.pem"
            ssl_certificate_verification => false
        }
}

I got error in logstash logs(logstash-plain.log) when I start logstash:

[2020-04-01T19:12:26,533][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-01T19:12:26,547][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.2"}
[2020-04-01T19:12:28,102][INFO ][org.reflections.Reflections] Reflections took 62 ms to scan 1 urls, producing 19 keys and 39 values 
[2020-04-01T19:12:28,724][WARN ][logstash.outputs.elasticsearch] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2020-04-01T19:12:28,994][ERROR][logstash.javapipeline    ] /http_client_builder.rb:99)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$build$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", 
.................Omit extra long exception stack...............
[2020-04-01T19:12:29,028][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-04-01T19:12:29,227][INFO ][logstash.runner          ] Logstash shut down.

But I don't know where I'm wrong from the log above，Can you find out what caused it?

Robo · April 1, 2020, 12:04pm

what will happen if you set ?
ssl_certificate_verification => true

chauncy1 · April 2, 2020, 12:39am

ssl_certificate_verification => true
Same error as I post, the following is the complete log：

[2020-04-02T08:35:49,595][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-02T08:35:49,626][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.2"}
[2020-04-02T08:35:51,519][INFO ][org.reflections.Reflections] Reflections took 126 ms to scan 1 urls, producing 19 keys and 39 values 
[2020-04-02T14:39:00,867][ERROR][logstash.javapipeline    ]  /http_client_builder.rb:99)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$build$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.CollectHandle.invokeExact_thunkArchetype_X(java/lang/invoke/CollectHandle.java:122)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.build_client(/opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb:238)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.RUBY$method$build_client$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.register(/opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:25)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:548)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:354)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:246)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:106)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:91)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$register.call(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt$INVOKER$i$0$0$register.gen)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:192)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:191)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.CollectHandle.invokeExact_thunkArchetype_X(java/lang/invoke/CollectHandle.java:122)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.maybe_setup_out_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:462)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$maybe_setup_out_plugins$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:204)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.run(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:146)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:105)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.FoldVoidHandle.invokeExact_thunkArchetype_X(java/lang/invoke/FoldHandle.java:156)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:811)"], :thread=>"#<Thread:0x88af2c5c run>"}
[2020-04-02T08:35:52,547][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-04-02T08:35:52,769][INFO ][logstash.runner          ] Logstash shut down.

Robo · April 2, 2020, 6:27am

Try to increase the log level of Logstash. Does the user logstash_internal have sufficient privileges (write, create_index, manage)? Is there any related record in ES log?

chauncy1 · April 2, 2020, 6:44am

I don't think it's problem of user privilege. I also get same error as last comment when I change user to elastic(superuser role).
The configuration is as follows：

output{
    elasticsearch{
        hosts => "53.99.41.73:9200"
        index => "monitor_message_flow"
        document_id => "%{mmfid}"
        user => "elastic"
        password => "password"
        ssl => true
        cacert => "/opt/es/logstash-mqiib/logstash-7.3.2/certs/cacert.pem"
        ssl_certificate_verification => true
    }
}

The log is the same as the last comment result:

[2020-04-02T14:38:58,615][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-02T14:38:58,630][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.2"}
[2020-04-02T14:39:00,062][INFO ][org.reflections.Reflections] Reflections took 60 ms to scan 1 urls, producing 19 keys and 39 values 
[2020-04-02T14:39:00,867][ERROR][logstash.javapipeline    ] /http_client_builder.rb:99)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$build$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.CollectHandle.invokeExact_thunkArchetype_X(java/lang/invoke/CollectHandle.java:122)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.build_client(/opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb:238)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.RUBY$method$build_client$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.register(/opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:25)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:548)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:354)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:246)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:106)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:91)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$register.call(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt$INVOKER$i$0$0$register.gen)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:192)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:191)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.CollectHandle.invokeExact_thunkArchetype_X(java/lang/invoke/CollectHandle.java:122)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.maybe_setup_out_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:462)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$maybe_setup_out_plugins$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:204)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.run(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:146)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:105)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.FoldVoidHandle.invokeExact_thunkArchetype_X(java/lang/invoke/FoldHandle.java:156)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:811)"], :thread=>"#<Thread:0xcb76dde2 run>"}
[2020-04-02T14:39:00,905][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-04-02T14:39:01,092][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-04-02T14:39:06,140][INFO ][logstash.runner          ] Logstash shut down.

And ElasticSearch is cluster with 3 node, kibana and logstash both are single node, is it related to this?

chauncy1 · April 2, 2020, 6:54am

Before I transform ELK to SSL and x_pack. My logstash.conf is as follwing:

output{
    elasticsearch{
        hosts => "53.99.41.73:9200"
        index => "monitor_message_flow"
        document_id => "%{mmfid}"
    }
}

It's working fine with only HTTP before. I just want to express the difference.

Robo · April 2, 2020, 7:01am

Try to increase log level of logstash. The configuration looks good.

chauncy1 · April 2, 2020, 7:48am

I set log.level: debug, and log is so long that it exceeds the word limit, I'll send it to you four times.

[2020-04-02T15:28:02,733][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/opt/es/logstash/logstash-7.3.2/modules/netflow/configur
ation"}
[2020-04-02T15:28:02,739][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0xff
fffffff0a88cb8 @directory="/opt/es/logstash/logstash-7.3.2/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2020-04-02T15:28:02,741][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/opt/es/logstash/logstash-7.3.2/modules/fb_apache/conf
iguration"}
[2020-04-02T15:28:02,742][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x
ffffffffb5050bab @directory="/opt/es/logstash/logstash-7.3.2/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2020-04-02T15:28:03,146][DEBUG][logstash.runner          ] -------- Logstash Settings (* means modified) ---------
[2020-04-02T15:28:03,146][DEBUG][logstash.runner          ] node.name: "DCDCA0000174.cn.svc.corpintra.net"
[2020-04-02T15:28:03,146][DEBUG][logstash.runner          ] *path.config: "/opt/es/logstash/logstash-7.3.2/config/logstash.conf"
[2020-04-02T15:28:03,146][DEBUG][logstash.runner          ] path.data: "/opt/es/logstash/logstash-7.3.2/data"
[2020-04-02T15:28:03,146][DEBUG][logstash.runner          ] modules.cli: []
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] modules: []
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] modules_list: []
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] modules_variable_list: []
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] modules_setup: false
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] config.test_and_exit: false
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] config.reload.automatic: false
[2020-04-02T15:28:03,147][DEBUG][logstash.runner          ] config.reload.interval: 3000000000
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] config.support_escapes: false
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] config.field_reference.parser: "STRICT"
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] metric.collect: true
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.id: "main"
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.system: false
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.workers: 4
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.batch.size: 125
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.batch.delay: 50
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.unsafe_shutdown: false
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.java_execution: true
[2020-04-02T15:28:03,148][DEBUG][logstash.runner          ] pipeline.reloadable: true
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] pipeline.plugin_classloaders: false
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] path.plugins: []
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] config.debug: false
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] *log.level: "debug" (default: "info")
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] version: false
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] help: false
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] log.format: "plain"
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] http.host: "127.0.0.1"
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] http.port: 9600..9700
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] http.environment: "production"
[2020-04-02T15:28:03,149][DEBUG][logstash.runner          ] queue.type: "memory"
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.drain: false
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.page_capacity: 67108864
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.max_bytes: 1073741824
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.max_events: 0
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.checkpoint.acks: 1024
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.checkpoint.writes: 1024
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.checkpoint.interval: 1000
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] queue.checkpoint.retry: false
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] dead_letter_queue.enable: false
[2020-04-02T15:28:03,150][DEBUG][logstash.runner          ] dead_letter_queue.max_bytes: 1073741824
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] slowlog.threshold.warn: -1
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] slowlog.threshold.info: -1
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] slowlog.threshold.debug: -1
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] slowlog.threshold.trace: -1
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] keystore.file: "/opt/es/logstash/logstash-7.3.2/config/logstash.keystore"
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] path.queue: "/opt/es/logstash/logstash-7.3.2/data/queue"
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] path.dead_letter_queue: "/opt/es/logstash/logstash-7.3.2/data/dead_letter_queue"
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] path.settings: "/opt/es/logstash/logstash-7.3.2/config"
[2020-04-02T15:28:03,151][DEBUG][logstash.runner          ] path.logs: "/opt/es/logstash/logstash-7.3.2/logs"
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.enabled: false
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.logstash.poll_interval: 5000000000
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.pipeline.id: ["main"]
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.elasticsearch.username: "logstash_system"
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.elasticsearch.hosts: ["https://localhost:9200"]
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.elasticsearch.ssl.verification_mode: "certificate"
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.management.elasticsearch.sniffing: false
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.monitoring.enabled: false
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.hosts: ["http://localhost:9200"]
[2020-04-02T15:28:03,152][DEBUG][logstash.runner          ] xpack.monitoring.collection.interval: 10000000000
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] xpack.monitoring.collection.timeout_interval: 600000000000
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.username: "logstash_system"
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.ssl.verification_mode: "certificate"
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] xpack.monitoring.elasticsearch.sniffing: false
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] xpack.monitoring.collection.pipeline.details.enabled: true
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] xpack.monitoring.collection.config.enabled: true
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] node.uuid: ""
[2020-04-02T15:28:03,153][DEBUG][logstash.runner          ] --------------- Logstash Settings -------------------
[2020-04-02T15:28:03,185][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-04-02T15:28:03,194][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.3.2"}
[2020-04-02T15:28:03,226][DEBUG][logstash.agent           ] Setting up metric collection
[2020-04-02T15:28:03,273][DEBUG][logstash.instrument.periodicpoller.os] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-04-02T15:28:03,529][DEBUG][logstash.instrument.periodicpoller.jvm] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-04-02T15:28:03,593][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"scavenge"}
[2020-04-02T15:28:03,596][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"global"}
[2020-04-02T15:28:03,610][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-04-02T15:28:03,621][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Starting {:polling_interval=>5, :polling_timeout=>120}
[2020-04-02T15:28:03,665][DEBUG][logstash.agent           ] Starting agent
[2020-04-02T15:28:03,706][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>["/opt/es/logstash/logstash-7.3.2/config/jdbc_bst.sql", "/opt/es/logstash/logstash-7.3.2/config/jvm.options", "/opt/es/logstash/logstash-7.3.2/config/log4j2.properties", "/opt/es/logstash/logstash-7.3.2/config/logstash-mq-appliance.conf", "/opt/es/logstash/logstash-7.3.2/config/logstash.yml", "/opt/es/logstash/logstash-7.3.2/config/pipelines.yml", "/opt/es/logstash/logstash-7.3.2/config/startup.options", "/opt/es/logstash/logstash-7.3.2/config/station_parameter.txt"]}
[2020-04-02T15:28:03,708][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/opt/es/logstash/logstash-7.3.2/config/oracle.conf"}
[2020-04-02T15:28:03,745][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>1}
[2020-04-02T15:28:03,751][DEBUG][logstash.agent           ] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
[2020-04-02T15:28:04,537][DEBUG][org.reflections.Reflections] going to scan these urls:
jar:file:/opt/es/logstash/logstash-7.3.2/logstash-core/lib/jars/logstash-core.jar!/

chauncy1 · April 2, 2020, 7:48am

[2020-04-02T15:28:04,596][INFO ][org.reflections.Reflections] Reflections took 58 ms to scan 1 urls, producing 19 keys and 39 values 
[2020-04-02T15:28:04,605][DEBUG][org.reflections.Reflections] expanded subtype co.elastic.logstash.api.Plugin -> co.elastic.logstash.api.Codec
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype co.elastic.logstash.api.Plugin -> co.elastic.logstash.api.Input
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype org.jruby.RubyBasicObject -> org.jruby.RubyObject
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype java.lang.Cloneable -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype org.jruby.runtime.builtin.IRubyObject -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype java.io.Serializable -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype java.lang.Comparable -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype org.jruby.runtime.marshal.CoreObjectType -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,606][DEBUG][org.reflections.Reflections] expanded subtype org.jruby.runtime.builtin.InstanceVariables -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,607][DEBUG][org.reflections.Reflections] expanded subtype org.jruby.runtime.builtin.InternalVariables -> org.jruby.RubyBasicObject
[2020-04-02T15:28:04,607][DEBUG][org.reflections.Reflections] expanded subtype co.elastic.logstash.api.Plugin -> co.elastic.logstash.api.Output
[2020-04-02T15:28:04,607][DEBUG][org.reflections.Reflections] expanded subtype co.elastic.logstash.api.Metric -> co.elastic.logstash.api.NamespacedMetric
[2020-04-02T15:28:04,607][DEBUG][org.reflections.Reflections] expanded subtype java.security.SecureClassLoader -> java.net.URLClassLoader
[2020-04-02T15:28:04,607][DEBUG][org.reflections.Reflections] expanded subtype java.lang.ClassLoader -> java.security.SecureClassLoader
[2020-04-02T15:28:04,608][DEBUG][org.reflections.Reflections] expanded subtype java.io.Closeable -> java.net.URLClassLoader
[2020-04-02T15:28:04,608][DEBUG][org.reflections.Reflections] expanded subtype java.lang.AutoCloseable -> java.io.Closeable
[2020-04-02T15:28:04,608][DEBUG][org.reflections.Reflections] expanded subtype java.lang.Comparable -> java.lang.Enum
[2020-04-02T15:28:04,608][DEBUG][org.reflections.Reflections] expanded subtype java.io.Serializable -> java.lang.Enum
[2020-04-02T15:28:04,608][DEBUG][org.reflections.Reflections] expanded subtype co.elastic.logstash.api.Plugin -> co.elastic.logstash.api.Filter
[2020-04-02T15:28:04,791][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"jdbc", :type=>"input", :class=>LogStash::Inputs::Jdbc}
[2020-04-02T15:28:04,902][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"plain", :type=>"codec", :class=>LogStash::Codecs::Plain}
[2020-04-02T15:28:04,918][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@id = "plain_9c3d0a83-451a-4bc3-821f-055eee7ff418"
[2020-04-02T15:28:04,919][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@enable_metric = true
[2020-04-02T15:28:04,919][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2020-04-02T15:28:04,936][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_user = "ESBV2"
[2020-04-02T15:28:04,937][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@last_run_metadata_path = "/opt/es/logstash/logstash-7.3.2/config/station_parameter.txt"
[2020-04-02T15:28:04,937][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_paging_enabled = true
[2020-04-02T15:28:04,937][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_page_size = 20000
[2020-04-02T15:28:04,937][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@statement_filepath = "/opt/es/logstash/logstash-7.3.2/config/jdbc_bst.sql"
[2020-04-02T15:28:04,938][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@tracking_column_type = "timestamp"
[2020-04-02T15:28:04,938][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@type = "jdbc"
[2020-04-02T15:28:04,938][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@record_last_run = true
[2020-04-02T15:28:04,938][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@use_column_value = true
[2020-04-02T15:28:04,938][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@tracking_column = "insertdbtime"
[2020-04-02T15:28:04,938][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@schedule = "*/2 * * * *"
[2020-04-02T15:28:04,940][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_password = <password>
[2020-04-02T15:28:04,940][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@clean_run = false
[2020-04-02T15:28:04,941][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_driver_library = "/opt/es/logstash/logstash-7.3.2/logstash-core/lib/jars/ojdbc6.jar"
[2020-04-02T15:28:04,941][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_connection_string = "jdbc:oracle:thin:ESBV2/U1c23GE_nHY=dtJKsM@//SCDCC0000072-scan:15016/ESBDBSRV"
[2020-04-02T15:28:04,941][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@id = "60d4c9c6cc19b49c4fe80e95f87bad40c4009e571ca2c6295a8f1f962ad3dfc4"
[2020-04-02T15:28:04,941][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_driver_class = "Java::oracle.jdbc.driver.OracleDriver"
[2020-04-02T15:28:04,941][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@enable_metric = true
[2020-04-02T15:28:04,951][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@codec = <LogStash::Codecs::Plain id=>"plain_9c3d0a83-451a-4bc3-821f-055eee7ff
418", enable_metric=>true, charset=>"UTF-8">
[2020-04-02T15:28:04,951][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@add_field = {}
[2020-04-02T15:28:04,952][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_validate_connection = false
[2020-04-02T15:28:04,952][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_validation_timeout = 3600
[2020-04-02T15:28:04,952][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@jdbc_pool_timeout = 5
[2020-04-02T15:28:04,952][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@sequel_opts = {}
[2020-04-02T15:28:04,952][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@sql_log_level = "info"
[2020-04-02T15:28:04,952][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@connection_retry_attempts = 1
[2020-04-02T15:28:04,953][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@connection_retry_attempts_wait_time = 0.5
[2020-04-02T15:28:04,953][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@parameters = {}
[2020-04-02T15:28:04,953][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@lowercase_column_names = true
[2020-04-02T15:28:04,953][DEBUG][logstash.inputs.jdbc     ] config LogStash::Inputs::Jdbc/@columns_charset = {}
[2020-04-02T15:28:04,983][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"ruby", :type=>"filter", :class=>LogStash::Filters::Ruby}
[2020-04-02T15:28:04,994][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@code = "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"
[2020-04-02T15:28:04,995][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@id = "ce0f7e31ee46098fc78128c727281ba3242fee1b5c5d73bfbffd7fa40e5ff588"
[2020-04-02T15:28:04,995][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@enable_metric = true
[2020-04-02T15:28:04,995][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_tag = []
[2020-04-02T15:28:04,995][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_tag = []
[2020-04-02T15:28:04,996][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_field = {}
[2020-04-02T15:28:04,996][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_field = []
[2020-04-02T15:28:04,996][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@periodic_flush = false
[2020-04-02T15:28:04,996][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@script_params = {}
[2020-04-02T15:28:04,996][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@tag_on_exception = "_rubyexception"
[2020-04-02T15:28:05,006][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@code = "event.set('@timestamp',event.get('timestamp'))"
[2020-04-02T15:28:05,006][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@id = "2a69a0884f7deb414cd7c49b516ae3b6f7f3ad3459879e252578b5741d7e34ea"
[2020-04-02T15:28:05,006][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@enable_metric = true
[2020-04-02T15:28:05,007][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_tag = []
[2020-04-02T15:28:05,007][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_tag = []
[2020-04-02T15:28:05,007][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_field = {}
[2020-04-02T15:28:05,007][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_field = []
[2020-04-02T15:28:05,007][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@periodic_flush = false
[2020-04-02T15:28:05,008][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@script_params = {}
[2020-04-02T15:28:05,008][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@tag_on_exception = "_rubyexception"
[2020-04-02T15:28:05,015][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@code = "event.set('creationtime', event.get('creationtime').time.localtime + 8*60*60)"
[2020-04-02T15:28:05,016][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@id = "cee0272f3780467d07e636111ca9d3c2ad83deb7afed8361d4cc5e6e6dcbbffd"
[2020-04-02T15:28:05,016][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@enable_metric = true
[2020-04-02T15:28:05,016][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_tag = []
[2020-04-02T15:28:05,016][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_tag = []
[2020-04-02T15:28:05,016][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_field = {}
[2020-04-02T15:28:05,017][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_field = []
[2020-04-02T15:28:05,017][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@periodic_flush = false
[2020-04-02T15:28:05,017][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@script_params = {}
[2020-04-02T15:28:05,017][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@tag_on_exception = "_rubyexception"
[2020-04-02T15:28:05,027][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@code = "event.set('insertdbtime', event.get('insertdbtime').time.localtime + 8*60*60)"
[2020-04-02T15:28:05,027][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@id = "ed2ae4db16c928eb2c244bd03a0b29d44e82d8b49714f7a625edb98148d71352"
[2020-04-02T15:28:05,028][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@enable_metric = true
[2020-04-02T15:28:05,028][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_tag = []
[2020-04-02T15:28:05,028][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_tag = []
[2020-04-02T15:28:05,028][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@add_field = {}
[2020-04-02T15:28:05,028][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@remove_field = []
[2020-04-02T15:28:05,028][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@periodic_flush = false
[2020-04-02T15:28:05,029][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@script_params = {}
[2020-04-02T15:28:05,029][DEBUG][logstash.filters.ruby    ] config LogStash::Filters::Ruby/@tag_on_exception = "_rubyexception"
[2020-04-02T15:28:05,039][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"mutate", :type=>"filter", :class=>LogStash::Filters::Mutate}

chauncy1 · April 2, 2020, 7:49am

[2020-04-02T15:28:05,051][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@remove_field = ["timestamp"]
[2020-04-02T15:28:05,052][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@id = "3d2480f21fcfbbd79ba596e01813ab91230d2a06cc31058377ad5f465e018f0b"
[2020-04-02T15:28:05,052][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@enable_metric = true
[2020-04-02T15:28:05,052][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@add_tag = []
[2020-04-02T15:28:05,052][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@remove_tag = []
[2020-04-02T15:28:05,052][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@add_field = {}
[2020-04-02T15:28:05,052][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@periodic_flush = false
[2020-04-02T15:28:05,053][DEBUG][logstash.filters.mutate  ] config LogStash::Filters::Mutate/@tag_on_failure = "_mutate_error"
[2020-04-02T15:28:05,056][DEBUG][logstash.plugins.registry] On demand adding plugin to the registry {:name=>"elasticsearch", :type=>"output", :class=>LogStash::Outputs::ElasticSearch}
[2020-04-02T15:28:05,084][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@id = "plain_91765bcf-1397-4f9a-ab1c-de321b223d88"
[2020-04-02T15:28:05,084][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@enable_metric = true
[2020-04-02T15:28:05,084][DEBUG][logstash.codecs.plain    ] config LogStash::Codecs::Plain/@charset = "UTF-8"
[2020-04-02T15:28:05,095][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ssl_certificate_verification = true
[2020-04-02T15:28:05,095][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@password = <password>
[2020-04-02T15:28:05,099][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@hosts = [//53.99.41.73:9200]
[2020-04-02T15:28:05,099][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@cacert = "/opt/es/logstash/logstash-7.3.2/certs/cacert.pem"
[2020-04-02T15:28:05,100][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@index = "monitor_message_flow"
[2020-04-02T15:28:05,100][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@id = "57e4b8dc00ec3880fd4f2eb74ff8a2026ef8672811d937dfdc11ad95c5184a37"
[2020-04-02T15:28:05,100][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@document_id = "%{mmfid}"
[2020-04-02T15:28:05,100][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@user = "elastic"
[2020-04-02T15:28:05,100][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ssl = true
[2020-04-02T15:28:05,100][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@enable_metric = true
[2020-04-02T15:28:05,101][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@codec = <LogStash::Codecs::Plain id=>"plain_91765bcf-1397-4f9a-ab1c-de321b223d88", enable_metric=>true, charset=>"UTF-8">
[2020-04-02T15:28:05,101][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@workers = 1
[2020-04-02T15:28:05,101][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@manage_template = true
[2020-04-02T15:28:05,101][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_name = "logstash"
[2020-04-02T15:28:05,101][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@template_overwrite = false
[2020-04-02T15:28:05,102][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@parent = nil
[2020-04-02T15:28:05,102][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@join_field = nil
[2020-04-02T15:28:05,102][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@upsert = ""
[2020-04-02T15:28:05,102][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@doc_as_upsert = false
[2020-04-02T15:28:05,102][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@script = ""
[2020-04-02T15:28:05,102][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@script_type = "inline"
[2020-04-02T15:28:05,103][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@script_lang = "painless"
[2020-04-02T15:28:05,103][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@script_var_name = "event"
[2020-04-02T15:28:05,103][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@scripted_upsert = false
[2020-04-02T15:28:05,103][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@retry_initial_interval = 2
[2020-04-02T15:28:05,103][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@retry_max_interval = 64
[2020-04-02T15:28:05,103][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@retry_on_conflict = 1
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@pipeline = nil
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_enabled = "auto"
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_rollover_alias = "logstash"
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_pattern = "{now/d}-000001"
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@ilm_policy = "logstash-policy"
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@action = "index"
[2020-04-02T15:28:05,104][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@sniffing = false
[2020-04-02T15:28:05,105][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@sniffing_delay = 5
[2020-04-02T15:28:05,105][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@timeout = 60
[2020-04-02T15:28:05,105][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@failure_type_logging_whitelist = []
[2020-04-02T15:28:05,105][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@pool_max = 1000
[2020-04-02T15:28:05,105][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@pool_max_per_route = 100
[2020-04-02T15:28:05,106][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@resurrect_delay = 5
[2020-04-02T15:28:05,106][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@validate_after_inactivity = 10000
[2020-04-02T15:28:05,106][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@http_compression = false
[2020-04-02T15:28:05,106][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@custom_headers = {}
[2020-04-02T15:28:05,138][DEBUG][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>"main"}
[2020-04-02T15:28:05,177][DEBUG][logstash.outputs.elasticsearch] Normalizing http path {:path=>nil, :normalized=>nil}

chauncy1 · April 2, 2020, 7:50am

[2020-04-02T15:28:05,427][ERROR][logstash.javapipeline    ] /http_client_builder.rb:99)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.http_client_builder.RUBY$method$build$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs/elasticsearch//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.CollectHandle.invokeExact_thunkArchetype_X(java/lang/invoke/CollectHandle.java:122)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.build_client(/opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb:238)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.RUBY$method$build_client$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java/lib/logstash/outputs//opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_elasticsearch_minus_10_dot_1_dot_0_minus_java.lib.logstash.outputs.elasticsearch.common.register(/opt/es/logstash/logstash-7.3.2/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:25)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:548)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:354)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.reg(org/logstash/config/ir/compiler/OutputStrategyExt.java:246)", "org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.register(org/logstash/config/ir/compiler/OutputStrategyExt.java:106)", "org.logstash.config.ir.compiler.OutputDelegatorExt.doRegister(org/logstash/config/ir/compiler/OutputDelegatorExt.java:91)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.AbstractOutputDelegatorExt$INVOKER$i$0$0$register.call(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt$INVOKER$i$0$0$register.gen)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:192)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792)", "org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minu
s_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.register_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:191)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$register_plugins$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.CollectHandle.invokeExact_thunkArchetype_X(java/lang/invoke/CollectHandle.java:122)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.maybe_setup_out_plugins(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:462)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$maybe_setup_out_plugins$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:204)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$start_workers$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.run(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:146)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$run$0$__VARARGS__(opt/es/logstash/logstash_minus_7_dot_3_dot_2/logstash_minus_core/lib/logstash//opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.VirtualHandle.invokeExact_thunkArchetype_L(java/lang/invoke/VirtualHandle.java:130)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.MutableCallSiteDynamicInvokerHandle.invokeExact_thunkArchetype_X(java/lang/invoke/MutableCallSiteDynamicInvokerHandle.java:64)", "opt.es.logstash.logstash_minus_7_dot_3_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start(/opt/es/logstash/logstash-7.3.2/logstash-core/lib/logstash/java_pipeline.rb:105)", "java.lang.invoke.DirectHandle.invokeExact_thunkArchetype_L(java/lang/invoke/DirectHandle.java:302)", "java.lang.invoke.BruteArgumentMoverHandle.invokeExact_thunkArchetype_X(java/lang/invoke/BruteArgumentMoverHandle.java:398)", "java.lang.invoke.FoldVoidHandle.invokeExact_thunkArchetype_X(java/lang/invoke/FoldHandle.java:156)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:811)"], :thread=>"#<Thread:0x598f6ef1 run>"}
[2020-04-02T15:28:05,468][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-04-02T15:28:05,499][DEBUG][logstash.agent           ] Starting puma
[2020-04-02T15:28:05,504][DEBUG][logstash.agent           ] Trying to start WebServer {:port=>9600}
[2020-04-02T15:28:05,506][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2020-04-02T15:28:05,521][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2020-04-02T15:28:05,521][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2020-04-02T15:28:05,522][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2020-04-02T15:28:05,526][DEBUG][logstash.agent           ] Shutting down all pipelines {:pipelines_count=>0}
[2020-04-02T15:28:05,528][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2020-04-02T15:28:05,540][DEBUG][logstash.api.service     ] [api-service] start
[2020-04-02T15:28:05,664][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-04-02T15:28:10,724][INFO ][logstash.runner          ] Logstash shut down.

Robo · April 2, 2020, 8:31am

what's the logstash.yml configuration (xpack.monitoring settings)?

chauncy1 · April 2, 2020, 8:53am

As log says: [2020-04-02T15:28:03,146][DEBUG][logstash.runner ] -------- Logstash Settings (* means modified) ---------
My logstash.yml only input one valid row: log.level: "debug", maybe it's default config?
And [2020-04-02T15:28:03,152][DEBUG][logstash.runner ] xpack.monitoring.enabled: false.

system · April 30, 2020, 8:53am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash not able to connect secured (ssl) Elastic search cluster Logstash	11	3906	June 19, 2018
Unable to Connect Logstash with Elastic Logstash elastic-stack-security	2	1152	August 3, 2021
Logstash can't establish pipeline with ssl Logstash elastic-stack-security	9	7896	August 20, 2019
Applying SSL to Elasticsearch and Kibana: Connection refused on Kibana Elasticsearch	55	1442	May 16, 2024
Client did not trust this server's certificate for the Cluster Elasticsearch elastic-stack-security	16	17102	June 10, 2019

Logstash enable ssl to connect ElasticSearch but cannot run logstash and get error

Related Topics