Logstash to elastic search TCP connection error

newelastic · April 29, 2022, 5:18pm

Hi,

I'm trying to connect to Elasticsearch using logstash and running into an TCP connection error.

Below is my input plugin for logstash

input {
  elasticsearch {
	hosts => ["https://esipAddress:9200"]
	index => "test_2201"
	user => "esadminuser"
	password => "pwd"
	ssl => true
	ca_file => "/etc/logstash/ssl/escert.pem"	
}
}

Below is TCP connection error.

E:\ELK\logstash-6.4.2\bin>logstash -f e:\ELK\logstash.conf
Sending Logstash logs to E:/ELK/logstash-6.4.2/logs which is now configured via log4j2.properties
[2022-04-29T10:13:08,903][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2022-04-29T10:13:09,280][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.2"}
[2022-04-29T10:13:10,610][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2022-04-29T10:13:10,836][INFO ][com.microsoft.azure.kusto.ingest.QueuedIngestClient] Creating a new IngestClient
[2022-04-29T10:13:10,852][INFO ][com.microsoft.azure.kusto.ingest.ResourceManager] Refreshing Ingestion Auth Token
[2022-04-29T10:13:10,883][INFO ][logstash.outputs.kusto   ] Going to recover old files in path
[2022-04-29T10:13:10,908][INFO ][logstash.outputs.kusto   ] Found 0 old file(s), sending them now...
[2022-04-29T10:13:11,309][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x72c22c15 run>"}
[2022-04-29T10:13:11,353][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2022-04-29T10:13:11,620][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2022-04-29T10:13:11,887][INFO ][com.microsoft.azure.kusto.ingest.ResourceManager] Refreshing Ingestion Resources
[2022-04-29T10:13:14,106][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::Elasticsearch password=><password>, ca_file=>"e:\\certs\\escert.pem", hosts=>["https://esIpAddress:9200"], index=>"test_2201", id=>"689c8935bd4a8544125f8cfee5e34acbd908ddcea32cab3d3b055403f1d48cd1", user=>"esadminuser", ssl=>true, enable_metric=>true, codec=><LogStash::Codecs::JSON id=>"json_8282870b-dc73-4b75-a2e6-cb646910eaac", enable_metric=>true, charset=>"UTF-8">, query=>"{ \"sort\": [ \"_doc\" ] }", size=>1000, scroll=>"1m", docinfo=>false, docinfo_target=>"@metadata", docinfo_fields=>["_index", "_type", "_id"]>
  Error: Failed to open TCP connection to https:0 (initialize: name or service not known)
  Exception: Faraday::ConnectionFailed
  Stack: org/jruby/ext/socket/RubyTCPSocket.java:137:in `initialize'
org/jruby/RubyIO.java:1154:in `open'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:885:in `block in connect'
org/jruby/ext/timeout/Timeout.java:149:in `timeout'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:883:in `connect'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:868:in `do_start'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:857:in `start'
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1409:in `request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:82:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:40:in `block in call'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/adapter/net_http.rb:32:in `call'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb:23:in `block in perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/faraday.rb:20:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/search.rb:183:in `search'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:200:in `do_run'
E:/ELK/logstash-6.4.2/vendor/bundle/jruby/2.3.0/gems/logstash-input-elasticsearch-4.2.1/lib/logstash/inputs/elasticsearch.rb:188:in `run'
E:/ELK/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:409:in `inputworker'
E:/ELK/logstash-6.4.2/logstash-core/lib/logstash/pipeline.rb:403:in `block in start_input'
[2022-04-29T10:13:15,173][ERROR][logstash.pipeline        ] A plugin had an unrecoverable error. Will restart this plugin.

Can someone throw if anything wrong with my plugin or missing anything here? TIA

aaron-nimocks · April 29, 2022, 5:25pm

I would start with the host parameter. Think this is the one where you don't put https in the host and use ssl => true instead. Try this.

input {
  elasticsearch {
    hosts => ["esipAddress:9200"]
    ssl => "true"
	index => "test_2201"
	user => "esadminuser"
	password => "pwd"
	ssl => true
	ca_file => "e:\certs\escert.pem"	
  }
}

newelastic · April 29, 2022, 6:14pm

Thank you for quick response. After removing https from hosts, I don't see any log or error. I'm running this logstash on my private VNET. I did look up into /usr/share/logstash/logs and don't see anymore logs.

The service is active and showing as running. However, I don't see any output Is there any other location to view the logstash progress? I set log.level: info FYI

My output plugin is as below

output {   
file {
      path => "/tmp/logstash/output.log"
   }
}

newelastic · April 29, 2022, 6:26pm

After restarting the service, I could see the logs in /mnt/logstash/logs

But, I'm seeing below error. I'm passing a valid SSL cert though.

Error: problem creating x509 Aux certificate java.io.IOException: unknown tag 13 encountered
Exception: Faraday::SSLError
stack: org/jruby/ext/openssl/SSLContext.java:402 in setup

aaron-nimocks · April 29, 2022, 6:29pm

Looks like a cert issue but I am not able to assist with that. If you don't get a response I would make a separate post asking for help with the cert now.

Badger · April 29, 2022, 7:33pm

Is there more to the stack trace? The "unknown tag 13 encountered" suggests it is expecting an ASN.1 byte stream but getting something else (possibly a base64 encoded stream).

The exception appears to be happening here. Going down that rabbit hole it appears to be building a certificate revocation list, which could be a call to (if I remember correctly) another web service, or even an LDAP server. The rest of the stack trace might help narrow that down.

system · May 27, 2022, 7:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issues connecting to logstash on windows server Logstash	2	1180	February 9, 2018
Log stash pipeline not connecting to Elasticsearch Logstash	4	1222	June 16, 2022
Using tcp plugin to parse logs from multiple sources Logstash	4	302	August 10, 2022
Logstash Not Connecting to Elasticsearch Logstash	10	14516	July 14, 2017
Logstash error during setup Logstash	18	1695	February 4, 2018

Logstash to elastic search TCP connection error

Related Topics