ES 7.14: can not get API key to work

I am trying to use an API key to authenticate using the API:

first the code then the stack trace...

      $logger.info "Try connecting to ES #{params}" if $logger
      begin
        @client =  Elasticsearch::Client.new params


 INFO  syslog-es-connect-loghost : Try connecting to ES {:hosts=>[{"host"=>"secmonprd08.its.auckland.ac.nz", "scheme"=>"https", "port"=>9200}], :request_timeout=>10, :randomize_hosts=>true, :api_key=>"..."}
Traceback (most recent call last):
	19: from /usr/local/tools/dev/siem_logging/bin/syslog-ng-es.rb:193:in `<main>'
	18: from /usr/local/tools/dev/siem_logging/bin/syslog-ng-es.rb:193:in `new'
	17: from /usr/local/tools/dev/siem_logging/lib/source/loghost.rb:36:in `initialize'
	16: from /usr/local/tools/dev/siem_logging/lib/source.rb:49:in `initialize'
	15: from /usr/local/tools/dev/siem_logging/lib/source.rb:92:in `open_es'
	14: from /usr/local/tools/dev/siem_logging/lib/source.rb:92:in `new'
	13: from /usr/local/tools/dev/common-library/lib/es_connection.rb:51:in `initialize'
	12: from /usr/local/tools/dev/common-library/lib/es_connection.rb:103:in `connect'
	11: from /usr/local/tools/dev/common-library/lib/es_connection.rb:103:in `new'
	10: from /var/lib/gems/2.7.0/gems/elasticsearch-7.15.0/lib/elasticsearch.rb:35:in `initialize'
	 9: from /var/lib/gems/2.7.0/gems/elasticsearch-7.15.0/lib/elasticsearch.rb:35:in `new'
	 8: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/client.rb:171:in `initialize'
	 7: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/client.rb:171:in `new'
	 6: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:60:in `initialize'
	 5: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:149:in `__build_connections'
	 4: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:149:in `map'
	 3: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:157:in `block in __build_connections'
	 2: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/http/faraday.rb:63:in `__build_connection'
	 1: from /var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:237:in `__full_url'
/var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:237:in `+': no implicit conversion of nil into String (TypeError)

lots of fiddling with the debugger established that Faraday was expecting a URL parameter and died when it was nil.

I then reworked the code to use these parameters:

{:hosts=>[{:host=>"https://secmonprd08.its.auckland.ac.nz:9200"}], :request_timeout=>10, :randomize_hosts=>true, :api_key=>"..."}

this time I got a client but when I tried @health = @client.cluster.health I got another crash.

More work in the debugger yeilded:

/var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:286:            response = block.call(connection, url)
(rdb:1) p url
"http://https://secmonprd08.its.auckland.ac.nz:9200:9200/"

which, unsurprisingly, causes a crash later on. However if I use the debugger:

(rdb:1) url =  "https://secmonprd08.its.auckland.ac.nz:9200"
"https://secmonprd08.its.auckland.ac.nz:9200"
...........
/var/lib/gems/2.7.0/gems/elasticsearch-transport-7.15.0/lib/elasticsearch/transport/transport/base.rb:344:          took     = (json['took'] ? sprintf('%.3fs', json['took']/1000.0) : 'n/a') rescue 'n/a'
(rdb:1) p json
{"name"=>"secmonprd08", "cluster_name"=>"test", "cluster_uuid"=>"mSgZQOoHRn-ttSYsjo60GA", "version"=>{"number"=>"7.14.0", "build_flavor"=>"default", "build_type"=>"deb", "build_hash"=>"dd5a0a2acaa2045ff9624f3729fc8a6f40835aa1", "build_date"=>"2021-07-29T20:49:32.864135063Z", "build_snapshot"=>false, "lucene_version"=>"8.9.0", "minimum_wire_compatibility_version"=>"6.8.0", "minimum_index_compatibility_version"=>"6.0.0-beta1"}, "tagline"=>"You Know, for Search"}

it works.

Any clues as to what is going on?

Sharp eyes may notice that I said that the ES version is 7.15 but the API version is 7.15. I noticed this while I was writing this up and went back and installed the 7.14 version of the gem and verified it performed the same.

I have had this code working fine for some time without https and the API key using the hash form of the parameters

Amazing what a nights sleep does to one's perceptions!

This morning I did what I should have done at the start and abstracted the problem down to a single line program:

require 'elasticsearch'

@client =  Elasticsearch::Client.new {:hosts=>[{"host"=>"secmonprd08.its.auckland.ac.nz", "scheme"=>"https", "port"=>9200}], :request_timeout=>10, :randomize_hosts=>true, :api_key=>"..."}

running this gives exactly the same crash as the before. Then I noticed that the keys if the hosts hash were stings not symbols! They were inserted into to the arguments hash from another hash which originates as a json file.

client =  Elasticsearch::Client.new hosts:[{host: "secmonprd08.its.auckland.ac.nz", scheme: "https", port: 9200}], :request_timeout=>10, :randomize_hosts=>true, :api_key=>"..."

works as expected.

I need to convert the keys of that hash to symbols before using it in parameters.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.