Logstash to elastic search TCP connection error

newelastic · April 29, 2022, 5:18pm

Hi,

I'm trying to connect to Elasticsearch using logstash and running into an TCP connection error.

Below is my input plugin for logstash

input {
  elasticsearch {
	hosts => ["https://esipAddress:9200"]
	index => "test_2201"
	user => "esadminuser"
	password => "pwd"
	ssl => true
	ca_file => "/etc/logstash/ssl/escert.pem"	
}
}

Below is TCP connection error.

E:\ELK\logstash-6.4.2\bin>logstash -f e:\ELK\logstash.conf
Sending Logstash logs to E:/ELK/logstash-6.4.2/logs which is now configured via log4j2.properties
[2022-04-29T10:13:08,903][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-04-29T10:13:09,280][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2022-04-29T10:13:10,610][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2022-04-29T10:13:10,836][INFO ][com.microsoft.azure.kusto.ingest.QueuedIngestClient] Creating a new IngestClient
[2022-04-29T10:13:10,852][INFO ][com.microsoft.azure.kusto.ingest.ResourceManager] Refreshing Ingestion Auth Token
[2022-04-29T10:13:10,883][INFO ][logstash.outputs.kusto   ] Going to recover old files in path
[2022-04-29T10:13:10,908][INFO ][logstash.outputs.kusto   ] Found 0 old file(s), sending them now...
[2022-04-29T10:13:11,309][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x72c22c15 run>"}
[2022-04-29T10:13:11,353][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-04-29T10:13:11,620][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2022-04-29T10:13:11,887][INFO ][com.microsoft.azure.kusto.ingest.ResourceManager] Refreshing Ingestion Resources
[2022-04-29T10:13:14,106][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Elasticsearch password=><password>, ca_file=>"e:\\certs\\escert.pem", hosts=>["https://esIpAddress:9200"], index=>"test_2201", id=>"689c8935bd4a8544125f8cfee5e34acbd908ddcea32cab3d3b055403f1d48cd1", user=>"esadminuser", ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_8282870b-dc73-4b75-a2e6-cb646910eaac", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"]>
  Error: Failed to open TCP connection to https:0 (initialize: name or service not known)
  Exception: Faraday::ConnectionFailed
  Stack: org/jruby/ext/socket/RubyTCPSocket.java:137:in `initialize'
org/jruby/RubyIO.java:1154:in `open'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:885:in `block in connect'
org/jruby/ext/timeout/Timeout.java:149:in `timeout'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:883:in `connect'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:868:in `do_start'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:857:in `start'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1409:in `request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:200:in `do_run'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:188:in `run'
E:/ELK/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
E:/ELK/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[2022-04-29T10:13:15,173][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.

Can someone throw if anything wrong with my plugin or missing anything here? TIA

aaron-nimocks · April 29, 2022, 5:25pm

I would start with the host parameter. Think this is the one where you don't put https in the host and use ssl => true instead. Try this.

input {
  elasticsearch {
    hosts => ["esipAddress:9200"]
    ssl => "true"
	index => "test_2201"
	user => "esadminuser"
	password => "pwd"
	ssl => true
	ca_file => "e:\certs\escert.pem"	
  }
}

newelastic · April 29, 2022, 6:14pm

Thank you for quick response. After removing https from hosts, I don't see any log or error. I'm running this logstash on my private VNET. I did look up into /usr/share/logstash/logs and don't see anymore logs.

The service is active and showing as running. However, I don't see any output Is there any other location to view the logstash progress? I set log.level: info FYI

My output plugin is as below

output {   
file {
      path => "/tmp/logstash/output.log"
   }
}

newelastic · April 29, 2022, 6:26pm

After restarting the service, I could see the logs in /mnt/logstash/logs

But, I'm seeing below error. I'm passing a valid SSL cert though.

Error: problem creating x509 Aux certificate java.io.IOException: unknown tag 13 encountered
Exception: Faraday::SSLError
stack: org/jruby/ext/openssl/SSLContext.java:402 in setup

aaron-nimocks · April 29, 2022, 6:29pm

Looks like a cert issue but I am not able to assist with that. If you don't get a response I would make a separate post asking for help with the cert now.

Badger · April 29, 2022, 7:33pm

Is there more to the stack trace? The "unknown tag 13 encountered" suggests it is expecting an ASN.1 byte stream but getting something else (possibly a base64 encoded stream).

The exception appears to be happening here. Going down that rabbit hole it appears to be building a certificate revocation list, which could be a call to (if I remember correctly) another web service, or even an LDAP server. The rest of the stack trace might help narrow that down.

system · May 27, 2022, 7:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.