LS-ES connection issue

Kanthasamyraja · February 20, 2019, 12:15pm

Hi,

Unable to start Logstash after xpack enabled.

Exception
Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://HOSTNAME:9200/_xpack'"}

I am able to login in browser http://HOSTNAME:9200/_xpack using the password mentioned in logstash.yml

xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: t0p.s3cr3t

cat conf.d/30-elasticsearch-output.conf
output {
elasticsearch {
user => "logstash_system"
password => "t0p.s3cr3t"
hosts => ["http://HOSTNAME:9200"]
manage_template => false
ssl => true
}
stdout { codec => rubydebug }
}

can anyone help?

Thanks

Badger · February 20, 2019, 12:30pm

Is there an error logged in the elasticsearch logs?

Kanthasamyraja · February 20, 2019, 12:32pm

Thanks @Badger for checking this.

No error message in elastic search.

followed the elastic document. Elastic search and Kibana up and running. created users and roles.

ikakavas · February 20, 2019, 1:49pm

ssl => true

doesn't make much sense if you are accessing elasticsearch over http

hosts => ["http://HOSTNAME:9200"]

Are you sure ?

Whats the output when you hit the _authenticate API with that user, i.e.

curl -ulogstash_system:t0p.s3cr3t -X GET http://HOSTNAME:9200/_xpack/security/_authenticate

Kanthasamyraja · February 20, 2019, 3:12pm

Thanks for your response.

Output
$ curl -ulogstash_system:t0p.s3cr3t -X GET http://HOSTNAME:9200/_xpack/security/_authenticate
{"username":"logstash_system","roles":["logstash_system"],"full_name":null,"email":null,"metadata":{"_reserved":true},"enabled":true}

Elasticsearch,Logstash and Kiban running on same machine.

ikakavas · February 20, 2019, 3:29pm

You should use a specific user for your elasticsearch output plugin that has the necessary permissions. Can you please try and follow the instructions in https://www.elastic.co/guide/en/logstash/6.6/ls-security.html#ls-http-auth-basic and create the necessary role and user and use that user for your configuration ?

Kanthasamyraja · February 20, 2019, 5:17pm

Hi @ikakavas

Thanks for your reply.

Created user and assigned almost all roles. Included my custom indexes as well.

POST _xpack/security/user/logstash_internal
{
"password" : "x-pack-test-password",
"roles" : [ "logstash_reader", "logstash_admin","logstash_system","logstash_writer"],
"full_name" : "Internal Logstash User"
}

PUT _xpack/security/user/logstash_internal/_enable
curl -ulogstash_internal:x-pack-test-password -X GET http://HOSTNAME:9200/_xpack/security/_authenticate

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [logstash_internal] for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [logstash_internal] for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

Restarted Elasticsearch and logstash.

Still same. could you please advice.

Thanks

ikakavas · February 20, 2019, 5:27pm

This is not exactly still the same.

I'm not sure what you mean here. logstash_writer doesn't need permissions to any of your indices. Can you please follow the instructions from the link I shared with you ?

This fails so I guess you either didn't do

POST _xpack/security/user/logstash_internal
{
"password" : "x-pack-test-password",
"roles" : [ "logstash_reader", "logstash_admin","logstash_system","logstash_writer"],
"full_name" : "Internal Logstash User"
}

as you said you did, or you entered another password.. It's really hard to follow what you are doing if you don't post exactly your steps and the outputs you get.

Also

why add all these roles to the logstash_internal user ?

Finally - this doesn't matter now as the logstash_internal user is not yet functional - you'd need to update your elasticearch output plugin configuration to use this user instead of logstash_system

Can you please make sure you follow the instructions in the documentation, or if you don't (why?), please add all the steps you take and all the outputs from the commands you run because otherwise we can't guess what might went wrong and where.

Kanthasamyraja · February 20, 2019, 5:51pm

I have deleted the user and recreated it like below.

POST _xpack/security/user/logstash_internal
{
"password" : "x-pack-test-password",
"roles" : [ "logstash_system"],
"full_name" : "Internal Logstash User"
}

{
"user" : {
"created" : true },
"created" : true}

enable

PUT _xpack/security/user/logstash_internal/_enable

{ }

curl -ulogstash_system:t0p.s3cr3t -X GET http://HOSTNAME:9200/_xpack/security/_authenticate

gave me this

{"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [logstash_internal] for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"unable to authenticate user [logstash_internal] for REST request [/_xpack/security/_authenticate]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

Updated output file

$ cat 30-elasticsearch-output.conf
output {
elasticsearch {
user => logstash_internal
password => x-pack-test-password
hosts => ["HOSTNAME:9200"]
manage_template => false
index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

Restarted Logstash. Got below error message.

[2019-02-20T18:48:24,477][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["main"], :poll_interval=>"5000000000ns"}
[2019-02-20T18:48:56,649][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Got response code '401' contacting Elasticsearch at URL 'http://HOSTNAME:9200/_xpack'"}
[2019-02-20T18:48:57,825][ERROR][logstash.configmanagement.elasticsearchsource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
[2019-02-20T18:48:57,856][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::LicenseChecker::LicenseError: Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.>, :backtrace=>["/usr/share/logstash/x-pack/lib/license_checker/licensed.rb:67:in `with_license_check'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:46:in `initialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:41:in `after_bootstrap_checks'", "org/logstash/execution/EventDispatcherExt.java:69:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:293:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:237:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}
[2019-02-20T18:48:58,072][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I didn't enable ssl. Using http only.
When trying in browser 'http://HOSTNAME:9200/_xpack' .Pop up asking for user name and password. entered logstash_internal and x-pack-test-password.Window keep coming and asking for user name and password

ikakavas · February 20, 2019, 6:26pm

You are still not following the documentation. Is there any reason for that ? Do you have some requirements that the given documentation is not satisfying?

Here you create logstash_internal. Why do you give them logstash_system role and not logstash_writer as instructed in the documentation? Have you created the logstash_writer role as instructed in the documentation?

you don't need to enable the newly created user, it's enabled by default.

This cannot happen. You show us a request that tries to authenticate as logstash_system with password t0p.s3cr3t and an error message that the user logstash_internal cannot be authenticated. You either shared the wrong command or the wrong output with us.

Please do share the logs from your elasticsearch node. I asked this in LS-ES connection issue - #4 by ikakavas before, but there might be something wrong with your node and the problem might well be there, instead of your logstash config.

You should do that first , or use the authenticate API, it's the same thing. If this doesn't work, there is no need to try to start logstash, it will not connect to Elasticsearch as it will use the same credentials.

In summary:

Check the Elasticsearch logs. I suspect there are some errors there.
I guess you have the password for some of the built in users since you can login to kibana and execute queries. Try executing GET /_cluster/state and GET /_cluster/health and share the response.
Share your security related configuration with us from elasticsearch.yml (filtering out any sensitive data of course )
Please use the documentation steps, unless otherwise necessary.

Kanthasamyraja · February 21, 2019, 10:05am

Hi,

Apologies for the copy paste error in previous comment.

$ cat elasticsearch.yml
path.data: /datavg/elasticsearch/data
path.logs: /datavg/elasticsearch/log
network.host: HOSTNAME
http.port: 9200
xpack.security.enabled: true
xpack:
  security:
    authc:
      realms:
        ldap1:
          type: ldap
          order: 0
          url: "ldaps://HOSTNAME:636"
          user_dn_templates:
            - ****
          group_search:
            base_dn: ***
          files:
            role_mapping: "/etc/elasticsearch/role_mapping.yml"
          unmapped_groups_as_roles: false

GET _cluster/settings

{
  "persistent" : {
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

GET /_cluster/health

{
  "cluster_name" : "elasticsearch",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 160,
  "active_shards" : 160,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 132,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 54.794520547945204
}

ikakavas · February 21, 2019, 10:26am

Please use the </> button to format your posts, it's otherwise really hard to read through your configuration. You can use the preview panel to see how it looks . It would be awesome if you can update your last post to do so.

You problem is that you have defined only an ldap realm and the native realm (the one which holds the logstash_internal user you are creating) is not available unless explicitly defined. So Elasticsearch tries to authenticate the logstash_internal user using LDAP and this fails as your LDAP server doesn't have any such user.

In addition to following the documentation that I have shared with you, you also need to add

native1:
  type: native
  order: 0

under realms in your elasticsearch.yml and also change the order of your ldap realm to 1.

Kanthasamyraja · February 21, 2019, 11:25am

Hi @ikakavas

Thanks for the information. Updated previous comment.

Updated elasticsearch.yml

curl command giving me error.

curl -ulogstash_internal:x-pack-test-password -X GET http://HOSTNAME:9200/_xpack/security/_authenticate

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"unknown cluster privilege [[manage_ilm, manage_index_templates, monitor]]. a privilege must be either one of the predefined fixed cluster privileges [all=[all],monitor_ml=[monitor_ml],manage_ingest_pipelines=[manage_ingest_pipelines],read_ccr=[read_ccr],monitor_watcher=[monitor_watcher],manage_index_templates=[manage_index_templates],manage_ccr=[manage_ccr],monitor=[monitor],manage_rollup=[manage_rollup],none=[none],manage_watcher=[manage_watcher],manage=[manage],manage_ml=[manage_ml],manage_pipeline=[manage_pipeline],monitor_rollup=[monitor_rollup],transport_client=[transport_client],manage_security=[manage_security],manage_saml=[manage_saml]] or a pattern over one of the available cluster actions"}],"type":"illegal_argument_exception","reason":"unknown cluster privilege [[manage_ilm, manage_index_templates, monitor]]. a privilege must be either one of the predefined fixed cluster privileges [all=[all],monitor_ml=[monitor_ml],manage_ingest_pipelines=[manage_ingest_pipelines],read_ccr=[read_ccr],monitor_watcher=[monitor_watcher],manage_index_templates=[manage_index_templates],manage_ccr=[manage_ccr],monitor=[monitor],manage_rollup=[manage_rollup],none=[none],manage_watcher=[manage_watcher],manage=[manage],manage_ml=[manage_ml],manage_pipeline=[manage_pipeline],monitor_rollup=[monitor_rollup],transport_client=[transport_client],manage_security=[manage_security],manage_saml=[manage_saml]] or a pattern over one of the available cluster actions"},"status":400}

"reason":"unknown cluster privilege [[manage_ilm, manage_index_templates, monitor]]. a privilege must be either one of the predefined fixed cluster privileges

Can start logstash once I remove the index previlges "manage_ilm". curl command also worked

curl -ulogstash_internal:x-pack-test-password -X GET http://HOSTNAME:9200/_xpack/security/_authenticate

{"username":"logstash_internal","roles":["logstash_writer"],"full_name":"Internal Logstash User","email":null,"metadata":{},"enabled":true}

Lostash showing the below error.

[2019-02-21T12:13:35,408][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigManagement::ElasticsearchSource::RemoteConfigError, :message=>"Empty configuration for pipeline_id: main", :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:99:in `get_pipeline'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:85:in `block in pipeline_configs'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:84:in `pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in `block in fetch'", "org/jruby/RubyArray.java:2481:in `collect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in `fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:150:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:118:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:107:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:362:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

Empty configuration for pipeline_id: main

Thanks

ikakavas · February 21, 2019, 11:45am

Looks like your authentication errors were all resolved once you followed the documentation.

Not sure what

 Could not fetch all the sources {:exception=>LogStash::ConfigManagement::ElasticsearchSource::RemoteConfigError, :message=>"Empty configuration for pipeline_id: main"

is caused by, I'll let people more familiar with Logstash take a look at it. In the meantime it would probably be helpful to post your logstash config, it will help people trying to assist you.

Kanthasamyraja · February 21, 2019, 12:07pm

Thanks @ikakavas for your time and help on this issue.

$ cat logstash.yml
path.data: /datavg/logstash/data
path.logs: /datavg/logstash/log
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: "x-pack-test-password"
xpack.monitoring.elasticsearch.url: "http://HOSTNAME:9200"
xpack.management.enabled: true
xpack.management.elasticsearch.username: logstash_admin_user
xpack.management.elasticsearch.password: t0p.s3cr3t
xpack.management.elasticsearch.url: "http://HOSTNAME:9200"

Kanthasamyraja · February 22, 2019, 10:14am

Hi @ikakavas

Created Pipeline in Logstash and in Kibana with name "main". Above mentioned error has gone now.

Now I am getting below error. Keeps printing.

[2019-02-22T10:51:44,248][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff  {:code=>403, :url=>"http://HOSTNAME:9200/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s"}

SSL enabled only in Kibana - self signed certificate

Logstash.yml

$ cat logstash.yml
path.data: /datavg/logstash/data
path.logs: /datavg/logstash/log
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: "x-pack-test-password"
xpack.monitoring.elasticsearch.url: "http://HOSTNAME:9200"
xpack.management.enabled: true
xpack.management.pipeline.id: ["main"]
xpack.management.elasticsearch.username: logstash_admin_user
xpack.management.elasticsearch.password: t0p.s3cr3t
xpack.management.elasticsearch.url: "http://HOSTNAME:9200"

elasticsearch.yml

$ cat elasticsearch.yml
path.data: /datavg/elasticsearch/data
path.logs: /datavg/elasticsearch/log
http.port: 9200
xpack.security.enabled: true
xpack:
  security:
    authc:
      realms:
        native1:
          type: native
          order: 0
        ldap1:
          type: ldap
          order: 1
          url: "ldaps://HOSTNAME:636"
		  ............
$

Kibana.yml

$ cat kibana.yml
server.host: "HOSTNAME"
server.name: "XXXXXXXX"
elasticsearch.url: "http://HOSTNAME:9200"
elasticsearch.preserveHost: true
elasticsearch.username: "kibana"
elasticsearch.password: "kibana"
server.ssl.enabled: false
server.ssl.enabled: true
server.ssl.key: /etc/kibana/ssl/XXXXXXXX.key
server.ssl.certificate: /etc/kibana/ssl/XXXXXXXX.cert
w1100411@ceala10649 (PQ_Agility_1-7.5):/etc/kibana 0
$

Kibana Error log shows this. Not sure both are related.

`
kibana[15422]: {"type":"error","@timestamp":"2019-02-22T09:45:34Z","tags":["connection","client","error"],"pid":15422,"level":"error","error":{"message":"TLS handshake timeout","name":"Error","stack":"Error: TLS handshake timeout\n    at TLSSocket._handleTimeout (_tls_wrap.js:596:22)\n    at Object.onceWrapper (events.js:313:30)\n    at emitNone (events.js:106:13)\n    at TLSSocket.emit (events.js:208:7)\n    at TLSSocket.Socket._onTimeout (net.js:422:8)\n    at ontimeout (timers.js:498:11)\n    at tryOnTimeout (timers.js:323:5)\n    at Timer.listOnTimeout (timers.js:290:5)"},"message":"TLS handshake timeout"}
`

Can you help here?

ikakavas · February 22, 2019, 10:41am

Hi,

Please share any related elasticsearch logs with us as these will help us understand better what the problem might be.

Also, this looks like you might have missed a step in https://www.elastic.co/guide/en/logstash/6.6/ls-security.html#ls-http-auth-basic . Can you verify that you created the logstash_writer role exactly as described in the first step of that doc?

Kanthasamyraja · February 25, 2019, 8:56am

Working for me now. I am using 6.5.4 version. In addition to the one mentioned in the document, gave permission for create. And updated missed out custom indices.
Thanks all for your help and time.

system · March 25, 2019, 8:56am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.