Unable to connect logstash with a secured cluster

Elastic Stack Logstash
1

Hi I'm using ELK 7.12.0 and I'm struggling to connect my logstash with my secured cluster, without xpack logstash send data to elasticsearch using twitter plugin, but when I enable xpack for elasticsearch and try to configure logstash for that I have the following error:

May 05 17:23:39 node2 logstash[13590]: [2021-05-05T17:23:39,671][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
May 05 17:23:39 node2 logstash[13590]: warning: thread "Ruby-0-Thread-10: :1" terminated with exception (report_on_exception is true):
May 05 17:23:39 node2 logstash[13590]: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'https://node1:9200/logstash'
May 05 17:23:39 node2 logstash[13590]:                     perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80
May 05 17:23:39 node2 logstash[13590]:              perform_request_to_url at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:317
May 05 17:23:39 node2 logstash[13590]:                     perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:304
May 05 17:23:39 node2 logstash[13590]:                     with_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:399
May 05 17:23:39 node2 logstash[13590]:                     perform_request at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:303
May 05 17:23:39 node2 logstash[13590]:                                Pool at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311
May 05 17:23:39 node2 logstash[13590]:                             exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:342
May 05 17:23:39 node2 logstash[13590]:              rollover_alias_exists? at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:364
May 05 17:23:39 node2 logstash[13590]:         maybe_create_rollover_alias at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:95
May 05 17:23:39 node2 logstash[13590]:                           setup_ilm at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:10
May 05 17:23:39 node2 logstash[13590]:                            register at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch.rb:275
May 05 17:23:39 node2 logstash[13590]:   setup_after_successful_connection at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:137
May 05 17:23:39 node2 logstash[13590]: [2021-05-05T17:23:39,781][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:317:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:304:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:399:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:303:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:311:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:342:in `exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/http_client.rb:364:in `rollover_alias_exists?'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:95:in `maybe_create_rollover_alias'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch/ilm.rb:10:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/outputs/elasticsearch.rb:275:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.8.2-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:137:in `block in setup_after_successful_connection'"]}
May 05 17:23:39 node2 logstash[13590]: [2021-05-05T17:23:39,826][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
May 05 17:23:39 node2 logstash[13590]: org.jruby.exceptions.SystemExit: (SystemExit) exit
May 05 17:23:39 node2 logstash[13590]:         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
May 05 17:23:39 node2 logstash[13590]:         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
May 05 17:23:39 node2 logstash[13590]:         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
May 05 17:23:40 node2 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
May 05 17:23:40 node2 systemd[1]: logstash.service: Failed with result 'exit-code'.

I followed the configuring security in logstash and here is what I did to set it up:

POST _xpack/security/role/logstash_writer
{
  "cluster": ["manage_index_templates", "monitor", "manage_ilm"], 
  "indices": [
    {
      "names": [ "logstash-*" ], 
      "privileges": ["write","create","create_index","manage","manage_ilm"]  
    }
  ]
}

POST _xpack/security/user/logstash_internal
{
  "password" : "my_password",
  "roles" : [ "logstash_writer"],
  "full_name" : "Internal Logstash User"
}

POST _xpack/security/role/logstash_reader
{
  "cluster": ["manage_logstash_pipelines"]
}

POST _xpack/security/user/logstash_user
{
  "password" : "my_password",
  "roles" : [ "logstash_reader", "logstash_admin"], 
  "full_name" : "Kibana User for Logstash"
}

and my twitter.conf file:

input {
    twitter {
        consumer_key => "I92pWf....."
        consumer_secret => "0oYfp3SawfVa....."
        keywords => ["corona", "covid", "covid19", "COVID19", "covid-19", "vaccin", "vaccination", "Vaccination"]
        oauth_token => "1359678032-46fTtM...."
        oauth_token_secret => "bcACovk....."
        #full_tweets => true
        #languages => [ "fr-FR", "en-US" ]
    }
}


output {
    elasticsearch {
        hosts => [ "https://node1:9200", "https://node2:9200" ]
        user => "logstash_internal"
        password => "my_password"
        ssl => true
        cacert => '/etc/logstash/config/certs/elasticsearch-ca.pem'
        ssl_certificate_verification => false
    }
    stdout { codec => rubydebug }
}

also the result of curl --cacert config/certs/elasticsearch-ca.pem -u logstash_internal 'https://node2:9200/_security/_authenticate' is:

{"username":"logstash_internal","roles":["logstash_writer"],"full_name":"Internal Logstash User","email":null,"metadata":{},"enabled":true,"authentication_realm":{"name":"default_native","type":"native"},"lookup_realm":{"name":"default_native","type":"native"},"authentication_type":"realm"}

Any idea please ?

2

The first thing I notice is that you haven't defined an index name in your output section?

3

Thanks for you reply, it's working without ssl/tls enabled and the index name been created is logstash-xxxxx, when I activate the tls layer it's just doesn't work, you mean adding index name will resolve the issue ?

4

If you have ILM enabled (which it will be by default, most likely) then you do not need to supply an index name.

You are getting a 403 when trying to set the template on the index. That suggests the role your user is assigned does not have the appropriate permission. I do not know enough about elasticsearch's permissions to say what permission you need.

5

Indeed @Badger, when I added the superuser role to logstash_internal user, logstash run successfully, I am trying now to find approriate role for logstash_internal to make it, because apparently I did the same thing from the doc but still not working! here is the roles I gave for logstash_internal user: ["write","create","create_index","manage","manage_ilm"]

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.